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ABSTRACT 



Governments and other stakeholders have become increasingly interested in assessing the s ki lls 
of their adult populations in order to monitor how well prepared they are to meet the challenges of the new 
information world. The current paper provides an overview of the conceptual framework for the 
assessment of numeracy developed for the OECD’s Programme for the International Assessment of Adult 
Competencies (PIAAC). This huilds on the numeracy framework developed for the Adult Literacy and 
Life Skills Survey (ALL). Numeracy is broadly defined and complemented with a definition of ‘numerate 
behaviour’. Lour facets of numerate behaviour are identified and described to guide the development of 
assessment tasks. 



RESUME 



Les gouvernements et autres parties prenantes de la region OCDE et en dehors s’interessent de plus en 
plus a revaluation des competences de leur population adulte, dans un objectif de suivi de leur etat de 
preparation pour faire face aux defis du nouveau monde de 1’ information. Le present document fournit une 
vue d’ensemble du cadre d’evaluation de 1’ aptitude au calcul du Programme international pour 1’evaluation 
des competences des adultes (PIAAC), a partir des cadres d’evaluation existants et des connaissances 
developpes par les enquetes « International Adult Literacy Survey » (lALS) menee dans les annees 1990 et 
« Adult Literacy and Life Skills Survey » (ALL) conduite en 2003 et 2006. La conceptualisation de 
1’ aptitude au calcul en tant que competence cle du cadre d’evaluation du PIAAC est detaillee et les 
differents contextes dans lesquels les individus y sont confrontes sont egalement decrits. En outre, le cadre 
d’evaluation presente le bareme a utiliser pour T evaluation directe de T aptitude au calcul au sein du 
PIAAC et explique comment ce bareme a ete defini, le distinguant d’ autres baremes d’ evaluations 
intemationales similaires. 
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PIAAC NUMERACY: CONCEPTUAL ERAMEWORK 



INTRODUCTION 



1. This document presents a framework for conceptualising numeracy and developing a scale for 
the direct assessment of adult numeracy as part of OECD’s Programme for International Assessment of 
Adult Competencies (PIAAC). Numeracy as viewed here refers to adults’ ability to access, use, interpret, 
and communicate mathematical information and ideas, in order to engage in and manage the mathematical 
demands of a range of situations in adult life. 

2. The conceptual framework and assessment issues discussed in this document were developed on 
the basis of several lines of work. This document builds on conceptual and assessment frameworks and 
cumulative wisdom developed in connection with prior surveys of adult skills, primarily the Adult Literacy 
and Life Skills Survey (ALL) and the International Adult Literacy Survey (lALS), but also surveys of 
school-age students (e.g., PISA, TIMSS). During preparation of this document by members of the PIAAC 
Numeracy Expert Group, a detailed review was conducted of professional and research literature in 
relevant areas such as on adult competencies, workplace skills, adult learning, or mathematics and statistics 
education. Ideas and feedback were also obtained from an international expert panel in 2006-2007 and all 
participating countries could react to drafts circulated for commentary during 2008. 

3. This framework is organised in six parts which cover separate but interrelated issues: 

Part 1 : A rationale for assessment of numeracy in PIAAC 

Part 2: Conceptual and theoretical foundations about adult numeracy 
Part 3: Lacets of numeracy 

Part 4: Scale development: Principles, constraints, implementation 

Part 5: Differences between PIAAC's numeracy and related constructs and scales 

Part 6: Summary and further reflections 
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PART 1: A RATIONALE FOR ASSESSMENT OF NUMERACY IN PIAAC 



1.1 Overview 

4. PIAAC is a policy-driven initiative intended to provide policy makers and key stakeholders at the 
national and international levels with information that can inform policy-setting and planning of social 
interventions and programs. PIAAC’s overarching goals (OECD, 2006) are twofold. Eirst, to identify and 
measure differences within and across countries in “literacy competencies for the information age - the 
interest, attitude, and ability of individuals to access, manage, integrate, and evaluate information, 
construct new knowledge, and communicate with others in order to participate effectively in the 
information age”. Second, to assess the relationship of adult competencies with economic and social 
outcomes believed to underlie both personal and societal success (e.g., earnings, employment, educational 
attainment, participation in further learning) and optionally with additional outcomes or processes at the 
individual level (e.g., health, social capital) or workplace level, and with transitions at key points over the 
lifespan, such as school-to-work and possibly other stages. 

5. The OECD planning paper states (OECD, 2006) that PIAAC is expected to provide reliable, 
valid, and valuable information to policy makers, differentiate the performance of low-scoring adults in 
each participating country, as well as include technology-based measures that tap into higher-order reading 
and thinking skills. As such, the design for the direct assessment in PIAAC incorporates computer-based 
measures of competencies, yet also use paper and pencil means for adults who are unfamiliar or 
uncomfortable with computers. 

6. PIAAC is further expected to enable continuity with and links to the two previous international 
adult assessments, the International Adult Literacy Survey (lALS) and the Adult Literacy and Life S ki lls 
Survey (ALL). Nineteen OECD countries’ participated in either lALS or ALL, with nine participating in 
both. Eor these countries, it is paramount that PIAAC’s direct assessment enables capitalisation on their 
previous investments and provides an indication of how adults’ competencies have changed since the 
previous measurement point(s). Equally important to maximizing countries’ past investments in adult 
learning, is adding value to lALS and ALL by seeking innovation by broadening what skills are measured 
as well as how they are measured. As the intent of PIAAC is to have its results linked to previous 
international adult assessments, PIAAC has been designed with a specification by OECD that 60% of the 
literacy and numeracy tasks will come from item pools used in ALL and lALS. As a result, this conceptual 
framework for assessing numeracy in PIAAC maintains conceptual and pragmatic links to the numeracy 
framework developed for ALL. 

7. Einally, OECD has expressed a desire that PIAAC’s direct assessment will be conceptually 
compatible with PISA. Although no direct statistical links or common items are necessarily expected 
between the two assessment programmes, such a linkage can provide policy-makers with information 
about the spectrum of competencies across different points of the lifespan, enable analyses of antecedents, 
consequences, and correlates of the distribution of competencies, and help to identify implications and 
relevant social interventions. 



1 Countries that participated in lALS include; Australia, Belgium (Flemish Community), Czech Republic, 

Canada, Denmark, Finland, Germany, Great Britain, Hungary, Ireland, Italy, Korea, Netherlands, New 
Zealand, Norway, Poland, Sweden, Switzerland, and the United States. Countries that participated in the 
first stage of ALL include: Bermuda, Canada, Italy, the Mexican State of Nuevo Leon, Norway, 
Switzerland, and the United States. ALU's second stage included Australia, Korea, Hungary, New Zealand, 
and the Netherlands. 
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1.2 Rationale For Assessing Numeracy In PIAAC 

8. ‘Numeracy’ is listed in the PIAAC overarching framework as a component of the hroad set of 
‘literacy competencies’. Yet, numeracy should he viewed as a key competency which is not subsumed 
under ‘literacy’ as this later term has been traditionally defined regarding reading, writing, and 
comprehending the meaning of text or communicating through textual means. While literacy and numeracy 
in the traditional sense have some linkages, numeracy is a broad construct with a life of its own and it has a 
central and often quite distinct role in adults’ lives. A later section reviews various perspectives which 
inform the conceptualisation of numeracy, leading to a definition of numeracy for PIAAC and description 
of facets or dimensions of numerate behavior. 

9. This framework is founded on the assumption that a direct assessment of numeracy in PIAAC is 
an essential and worthwhile undertaking (Willms, 2006; Murray, 2006), for four separate but related 
reasons: 

a) Numeracy is essential for adults and for the societies in which they live. Basic computational or 
mathematical knowledge has always been considered as part of the fundamental skills that adults 
need to possess to function well and be able to accomplish various goals in their everyday, work, 
and social life. Societies now present increasing amounts and wider range of information of a 
quantitative nature to citizens from all walks of life, in diverse contexts such as regarding health 
risk factors, school performance, or financial planning and insurance purchasing, to name just a 
few. As workplaces are becoming more concerned with involving all workers in improving 
efficiency and quality, the importance of numeracy skills is growing. Numeracy-related s ki lls 
have been shown to be a key factor in labor market participation, sometimes even more so than 
literacy skills. Adults with lower skills in numeracy and literacy are more likely to be 
unemployed or require social assistance. Further, some numeracy skills are deemed essential for 
post-secondary education in many areas, including but not limited to hard sciences, engineering 
and technology. (Jones, 1995; Murnane, Willett & Levy, 1995; Hoyles, Wolf, Molyneux-Hodson, 
& Kent, 2002; Coulombe, Tremblay, & Marchand, 2004; Desjardins, Murray, Clermont & 
Werquin, 2005). 

b) Public policy in most countries includes separate investments in literacy and numeracy. The 
separate acquisition of skills in these two fundamental areas is emphasised throughout both 
primary and secondary school systems, and in adult education or nonformal learning schemes. 
Countries expect that investment in literacy and numeracy will increase citizens’ ability to act 
independently towards their own progress and income security, thereby reducing future social 
expenditures as well as contributing to citizens’ participation in economic and social life in an 
information-laden society (European Commission, 1996). Numeracy has been shown to be 
associated with future learning which is important for re-training and for upgrading of skills as 
industries evolve (Marr & Hagston, 2007). The demands for higher numeracy performance will 
affect the employability of the labor force. 

c) The policy and program responses are different for numeracy than for literacy. Efforts to 
improve literacy and numeracy levels of specific population groups are not implemented via the 
same mechanisms — they often require different experts, resources, and learning systems because 
of differences in the underlying knowledge components and learning trajectories. It is vital that 
nations have information about their workers’ and citizens’ numeracy, independently of other 
competency areas, in order to evaluate the human capital available for advancement, to plan 
school-based and lifelong learning opportunities, and to better understand the factors that affect 
citizens' acquisition and usage of numeracy (Johnston, & MacGuire, 2005). 
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d) Numeracy skill levels are not measured well by literacy measures. It is not possible to represent 
the numeracy levels in a population via people’s performance on literacy measures that examine 
how well people read, process, and comprehend various types of texts and documents, or 
communicate about such texts. As explained later in more detail, numeracy involves, among 
other things, the handling of arithmetical processes, understanding of proportions and 
probabilistic ideas, understanding of numerical, geometric and graphical types and 
representations of quantitative information, critical interpretation of statistical or mathematical 
messages, ability to solve various types of quantitative problems, and other elements or processes 
that bear little relation to what is subsumed by literacy measures. (Cohen, 2000; Gal, van 
Groenestijn, Manly, Schmitt, & Tout, 2005) 

10. It follows that a direct assessment of numeracy in PIAAC can provide policy makers and other 
stakeholders with a sound basis for evaluating the distribution of the actual numeracy competence in the 
adult population. 



PART 2. CONCEPTUAL AND THEORETICAL EOUNDATIONS 



11. The conceptualisation of ‘numeracy’ in an international context is a challenging undertaking. 
Like literacy, the term numeracy has multiple meanings across countries and languages. In some countries 
the term numeracy relates to basic skills which school children are expected to acquire as a prerequisite to 
learning formal mathematics at higher grades. In other countries the term numeracy encompasses a broad 
range of skills, knowledge and dispositions that adults should possess but it does not necessarily relate to 
formal schooling (Baker & Street, 1994; NRDC, 2006). Finally, some countries do not even have a word 
such as numeracy; therefore, as part of educational or policy-oriented discourse in such countries, experts 
or translators either had to invent a special new word for it (e.g., ‘Numeratie’ in Canada, ‘Numeralitet’ in 
Denmark), or use other phrases such as “mathematical literacy”, “functional mathematics”, or terms 
equivalent to “computational ability”. Such diversity in terminology, or the lack of an accepted term with 
which policy-makers feel comfortable, can complicate the communication with and among policy makers 
interested in PIAAC. 

12. The range of meanings attached to the term numeracy and the lack of an equivalent term across 
languages may create miscommunications or gaps in expectations regarding what will be measured by a 
numeracy scale in PIAAC. This can affect the perceived policy relevance of a numeracy scale. Thus, 
attention has to be given to making sure that discussions regarding numeracy assessment in PIAAC are 
based on a consensus about the scope of the term and recognition of its centrality in a wide range of adult 
life circumstances. 

13. However, it must be remembered that what will be measured by a numeracy assessment scale is 
jointly determined by two interrelated factors - by (1) a conceptual scheme describing numeracy and its 
elements, and (2) by an assessment scheme describing how the general conceptualisation of numeracy is 
operationalised and manifested in the nature and range of tasks used in the assessment scale and the mode 
of administration and scoring. Of course, the conceptualisation and assessment of adult numeracy involves 
many questions, such as: What are the key numeracy tasks which adults have to face in their lives? What 
facets or sub-domains are subsumed under ‘adult numeracy’? What are the differences and commonalities 
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between numeracy and related key constructs such as Quantitative Literacy which was assessed in lALS or 
Mathematical Literacy which is assessed in PISA? 

14. This part is organised in three sections. In section 1, the notion of “competence” as defined by 
OLCD is outlined. In section 2, the contexts and situations in adults’ lives which require numeracy are 
reviewed. Section 3 examines perspectives on the meaning of numeracy and prior definitions and 
conceptualisations of numeracy, leading to a definition of numeracy for PIAAC. Part 3 which follows 
further examines the dimensions of numerate behavior, including contexts, expected responses, content 
areas of mathematical information and ideas, and representations, as a way of operationalizing the 
numeracy construct for scale development. It also discusses enabling processes, both cognitive and non- 
cognitive or dispositional, which underlie numerate behavior. Subsequent parts outline principles for 
assessment of numeracy in PIAAC, and comment on differences and commonalities between PIAAC's 
numeracy and related constructs assessed in lALS and PISA. 

2.1 Numeracy as a competence 

15. The conceptualisation of numeracy in an assessment program which focuses on “literacy 
competencies for the information age” has to be congruent with the broader notion of “competence”. 
Within OLCD, prior work on the Definition and Selection of Competencies project (DeSeCo; see Rychen 
& Salganic, 2003) has defined competence as “the ability to meet individual or social demands 
successfully, or to carry out an activity or task”. The DeSeCo view, which was adopted by OLCD and also 
informed the design of assessment scales for PISA, places at the forefront how individuals function in the 
face of external demands that may stem from a personal or social context of action. DeSeCo (2002: 8-9) 
conceptualises competencies as internal mental structures, i.e., abilities, capacities or dispositions 
embedded in the individual:. 

Each competence is built on a combination of interrelated cognitive and practical skills, 
knowledge (including tacit knowledge), motivation, value orientation, attitudes, emotions, and 
other social and behavioral components that together can be mobilised for effective action. 
Although cognitive skills and the knowledge base are critical elements, it is important not to 
restrict attention to these components of a competence, but to include other aspects such as 
motivation and value orientation. 

16. Lurther, DeSeCo (2002:7) argues that the terms “skills” and “competencies” are not synonyms. 
Skills designate an “ability to perform complex motor and/or cognitive acts with ease, precision, and 
adaptability to changing conditions”, while competence designates “a complex action system 
encompassing cognitive skills, attitudes and other non-cognitive components”. 

17. The conceptualisation of numeracy discussed below, which is based on a review of scholarly 
literature and research findings, operates on two levels. It relates to numeracy as a construct describing a 
competence as defined above, and to numerate behavior which is the way a person’s numeracy is 
manifested in the face of situations or contexts which have mathematical elements or carry information of 
a quantitative nature. In this way, inferences about a person’s numeracy are possible through analysis of 
performance on assessment tasks designed to elicit numerate behavior. In congruence with the above view 
of a competence, numeracy will be described as comprised both of cognitive elements (i.e., various 
knowledge bases and skills) as well as non-cognitive or semi -cognitive elements (i.e., attitudes, beliefs, 
habits of mind, and other dispositions) which together shape a person’ s numerate behavior. 
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2.2 Contexts and demands for numeracy 

18. Once a view of numeracy as a competence as defined above is adopted, a discussion of what is 
encompassed by numeracy (and numerate behavior) has to start by identifying the nature of the contexts 
which contain mathematical^ elements, or which include information of a quantitative nature, that adults 
face and which pose demands with which they have to cope. This in turn provides the basis for describing 
the knowledge elements and supporting processes which enable adults to cope with real-world numeracy 
tasks (Ginsburg, Manly & Schmitt, 2006), and can later help to form a road map which can guide the 
design and selection of tasks for inclusion in the numeracy assessment in PIAAC. 

19. The literature pertaining to the uses of numeracy in the real world can be divided into three 
strands: literature on the roles of literacy and numeracy in adults’ lives, on the mathematical demands of 
workplace and functional settings, and on educational perspectives on mathematical needs of school 
graduates and citizens. These areas are certainly intertwined but also offer complementary ideas, hence 
each is reviewed separately below. 

2.2.1. The roles of literacy and numeracy in adults’ lives. 

20. The purposes served by adults’ numeracy may parallel those served by adults' literacy, and 
further, people's numeracy may at times relate to or even depend in part on literacy skills or other lifeskills. 
Work to describe the purposes served by adults' literacy and numeracy skills has been conducted in several 
countries. In Australia, for example, Kindler et al., (1996) reported on four such purposes: literacy for self- 
expression, literacy for practical purposes, literacy for knowledge, and literacy for public debate. In the 
USA, the National Institute for Literacy has sponsored efforts to define critical skill areas. As part of its 
Equipped for the Euture initiative, four broad types of purposes were identified (Stein, 1995): Literacy for 
access and orientation in the world, literacy as voice to one’s ideas and opinions, literacy for independent 
action, solving problems and making decisions as a parent, citizen and worker, and literacy as a bridge to 
further learning and to keep up with a rapidly changing world. 

21. Work has also been done in adult education contexts to identify different purposes and functions 
of using mathematical knowledge. In Australia, for example, one key project (Kindler et al. 1996), pointed 
to four broad categories regarding the uses of numeracy: Numeracy for practical purposes addresses 
aspects of the physical world that involve designing, making, and measuring. Numeracy for interpreting 
society relates to interpreting and reflecting on numerical and graphical information in public documents 
and texts. Numeracy for personal organisation focuses on the numeracy requirements for personal 
organisational matters involving money, time and travel. Numeracy for knowledge describes the 
mathematical skills needed for further study in mathematics, or other subjects with mathematical 
underpinnings or assumptions. 

22. A scheme developed by Steen (1990), a noted mathematics educator, outlines five dimensions of 
numeracy: 

• Practical, focused on mathematical and statistical knowledge and skills that can be put to 
immediate use to cope with tasks in daily life 



2 The term “mathematical” is used here as inclusive of situations where statistical or probabilistic 

information may appear or where statistical thinking or statistical literacy are required as well. Such usage 
is made for brevity and convenience only. It is acknowledged that statistics is not a branch of mathematics, 
and that statistical reasoning and statistical literacy have unique elements, concepts and processes which 
are not mathematical in nature (Moore &. Cobb, 2000). 
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• Professional, focused on the mathematical skills required in specific jobs 

• Civic, focused on benefits to society 

• Recreational, related to the role of mathematical ideas and processes in games, puzzles, sports, 
lotteries, and other leisure activities 

• Cultural, concerned with mathematics as a universal part of human culture (and related to 
appreciation of mathematical aspects such as in cultural or artistic artifacts) 

23. Overall, the purposes regarding literacy and numeracy appear to agree and suggest that adults 
need to be able to apply their numeracy and literacy skills to tasks with a social or personal purpose in both 
informal and more formal contexts (NRDC, 2006). Such perspectives supplement Bishop’s (1988) 
proposal that there are six modes of mathematical actions that are common in all cultures and pertain both 
to children and adults: counting, locating, measuring, designing, playing and explaining. 

2.2.2. Numeracy in the workplace and in functional settings. 

24. Mathematical and statistical skills that are important in adults’ work have been described in 
large-scale efforts to define "core skills" or "key competencies" that workers should have, usually in 
response to the need to maintain economic competitiveness and improve employability of adults and 
school graduates. In addition, several projects looked specifically at the mathematical skills of workers in a 
range of occupational groups or workplace clusters. 

25. Basic computational knowledge has always been considered as part of the fundamental skills that 
adults need to possess, but recent skills frameworks claim that workers need to possess a much broader 
range of mathematical skills. Examples exist in many countries and the following selective description 
from the United States is indicative of the nature of such efforts. Eollowing earlier research by a task force 
of the American Society of Training and Development (Camevale, Gainer, & Meltzer, 1990), the U.S. 
Secretary of Labor’s Commission on Achieving Necessary Skills (SCANS) (Packer, 1997) has 
differentiated between mastery of basic arithmetical skills and much broader and flexible understanding of 
principles and underlying ideas subsumed under the notion of mathematical skills (SCANS, 1991, p. 83): 

• SCANS arithmetical skills: Performs basic computations; uses basic numerical concepts such as 
whole numbers and percentages in practical situations; makes reasonable estimates and 
arithmetic results without a calculator; and uses tables, graphs, diagrams and charts to obtain or 
convey quantitative information. 

• SCANS mathematical skills: Approaches practical problems by choosing appropriately from a 
variety of mathematical techniques; uses quantitative data to construct logical explanations for 
real world situations; expresses mathematical ideas and concepts orally and in writing; and 
understands the role of chance in the occurrence and prediction of events. 

26. Based on a later survey of employers, industry trainers, and educators, among others, Eorman & 
Steen (1999) similarly argued that quantitative skills desired by employers are much broader than mere 
facility with the mechanics of addition, subtraction, multiplication, and division and familiarity with basic 
number facts; they also include some knowledge of statistics, probability, mental computation strategies, 
some grasp of proportional reasoning or modeling relationships, and broad problem-solving and 
communication skills about quantitative issues. 
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27. Work on mathematical skills and their use in specific workplaces has been conducted over the 
last decade in both the manufacturing and service sectors in several countries such as the UK, Australia, 
USA, and others (e.g., Buckingham, 1997; Bessot & Ridgway, 2000; Hoyles et ah, 2002; Fitzsimons, 
2005; Skills Australia, 2005). Overall, these studies complement the SCANS study and suggest that 
employees need to possess a range of specific mathematical skills or knowledge, such as the following key 
(but not the only) examples: 

• skills in both fast and accurate computations but also estimation, and knowing when each skill is 
required and why 

• ability to deal with proportions and percents 

• understanding measurement concepts and procedures 

• working with or creating simple formulas 

• a sense for the use of models and modeling in foreseeing future needs 

• understanding of basic statistical concepts and displays 

28. In addition, on a broader and less technical level, these studies argue that workers need to be able 
to make decisions in the face of uncertainty in real situations, prioritise actions and make choices regarding 
the approach to handling different tasks, depending on changing external demands. As well, there is a need 
for workers to be able to communicate with other workers or clients or understand written documentation 
(e.g., through text or with tables, charts, and graphs) about issues such as quantities, schedules, variation 
over time, results of quantitative projections, or analysis of different courses of action in this regard. Such 
findings echo the earlier distinctions made by the SCANS analysis between the need to attend both to basic 
arithmetical skills and more elaborate and complex mathematical skills in the workplace, but also highlight 
some areas where specific literacy and communication skills are intertwined with numeracy s ki lls. 

29. An important research literature has also accumulated over the last decades regarding the ways in 
which people use mathematical skills or cope with mathematical tasks in both formal (i.e., school-based) 
and informal (i.e., everyday, workplace) contexts (e.g., Rogoff & Lave, 1984; Resnick, 1987; Saxe, 1988; 
Carraher, Schliemann, & Carraher, 1988; Scribner & Sachs, 1991; Nunes, 1992; Presmeg, 2007). While 
too complex to discuss in detail here (see Greeno, 2003, for one of several reviews of this literature), 
among other things these studies highlight the situatedness of mathematical knowledge used in functional 
contexts and the need for actors in different contexts to develop situation-specific mathematical procedures 
and know-how. Further, numerous researchers (e.g., Straesser, 2003; Wedege, 2003; Williams & Wake, 
2007) have argued, based on ethnographic analyses of workers’ activities in diverse industries, that 
important portions of the mathematical activities at work are made “invisible” to occasional observers as 
well as to the workers themselves, or are disguised as nonmathematical. Various factors have been posited 
as causing this phenomenon, such as the encapsulation of many mathematical activities into routines or 
automated procedures; the use of tools and instruments or information technology (e.g., spreadsheets); the 
normative use of job-specific linguistic terms that are different than traditional school terms; or the division 
of labor among different workers. 

30. Based on such and related findings, many projects have argued that mathematical skills as used in 
the workplace are often different and broader in scope than what is traditionally taught in school 
mathematics, but also take on different forms depending on the specific work context (Marr, & Hagston, 
2007). Overall, the above suggests that what employers, training and employment specialists, and 
researchers know about the mathematical or statistical demands of different occupations may be 
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incomplete. Further, projections of future skills demands usually focus on shifts in demand for workers 
with given skills, e.g., how many more engineers, technicians, or call-center operators will he required with 
a given mixture of currently-defined skills (Karoly, 2007), not on the changing future numeracy skills that 
workers will require. 

31. Most sources discuss future skills required in occupational or joh-market sectors, not to skills 
required in family, civic, or community contexts. Yet, with the accessibility of high-speed internet 
connections in homes, more adults are able to connect to, search, and make use of an increasing array of 
information sources, including many with quantitative components, such as regarding health, personal 
finances, comparative shopping, sports, education, official statistics, and more. Many service organisations 
increasingly open more opportunities for customers not only to access information, but also interact and 
take action, through Internet websites and other technology-based devices (e.g., ATM machines, 'smart' 
cellular phones, GPS navigation systems), thus involving adults in new types of transactions and activities. 
These changes are important because they blur the distinction between "work" and "non-work" situations 
and their skill demands in the area of numeracy. 

32. Given the above, the conceptualisation of numeracy for PIAAC was derived with reference to the 
types of numeracy demands as depicted earlier in this subsection. Further, a working assumption has been 
made that it is not feasible to employ assessment items that are very workplace-specific (e.g., couched in 
the context of a single workplace or occupation) because mathematics or statistics as used in this context 
may not be visible or familiar to most other adults (Hoyles et ah, 2002). 

2.2.3. Educational perspectives on numeracy and informed civic participation. 

33. A growing dialogue about the goals and impact of mathematics education in schools has 
intensified in recent years. This is in part due to economic pressures and industry expectations on the one 
hand, but also due to the realisation that mathematical knowledge and skills serve multiple and separate 
gateway functions on the other hand. Specifically, mathematical competencies affect chances of entry into 
key occupational tracks (mainly in science, technology, and economics) and may affect employability and 
labor-force participation, underlie some important aspects of civic participation, and may impact on the 
possibilities of certain population groups for social equality and mobility. While the dialogue about these 
issues admittedly overlaps to some extent the points raised earlier in discussing the roles of literacy and 
numeracy in society, it is worth elaborating upon because it brings forward some additional points and 
broadens the understanding of contexts where demands on adults’ numeracy exist. 

34. Various arguments have been forwarded over the last few decades to support a broadening of the 
conceptions regarding the mathematical skills and knowledge that school graduates should possess, and the 
ways in which learned knowledge serves adults (Frnest, 2004). Educators working both with school 
students and adults increasingly aim to assist learners in developing mathematical concepts and skills in 
ways that are personally meaningful but also functional. Such approaches usually assume that there is often 
more than one right way to cope with a real-world functional task, and that adults require access to a 
repertoire of strategies for solving functional problems. Adults’ personal methods of using mathematics are 
encouraged and valued. This is often a significant difference from traditional (pre-reform) school-based 
mathematics teaching, within which school students were often expected to solve a problem following the 
one correct method or algorithm, introduced by the teacher. 

35. Several decades ago, ideas already began emerging in different countries that since mathematics 
is an essential aspect of society, mathematics education in schools should be derived from or prepare 
learners for broad real-life situations in family, work, community, and other contexts (National Council of 
Teachers of Mathematics, 2000; Willis, 1990), beyond employers’ desire to focus mostly on practical or 
job-specific numeracy skills. Two early influential examples are the recommendations of the Cockcroft 
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Committee in the UK (Department of Education and Science/Welsh Office, 1982), and Ereudenthal’s work 
in the Netherlands which has led to the Realistic Mathematics Education movement (Heuvel-Panhuizen & 
Gravemeijer, 1991). Over the last two decades, various countries (e.g., Australia, UK) have adopted adult 
education frameworks which give explicit attention to numeracy skills. Eor example, in the UK, the Adult 
Numeracy Core Curriculum aims to move learners through up to five levels of demand, where the 
expectation is that at the highest one (level 2), an adult: 

Understands mathematical information used for different purposes and can independently select 
and compare relevant information from a variety of graphical, numerical and written material — 
e.g. compare data using mean and median, work out discounts as fractions and percentages of 
amounts, work out distances and lengths from scale drawings (Gillespie, 2007: p.4). 

36. Educators have also paid much attention to the importance of quantitative literacy in civic and 
social contexts, and argued that mathematical knowledge is a crucial part of a common fabric of 
communication indispensable for modem civilised society, in part because it is the language of science and 
technology. Thus, it has been claimed (National Research Council, 1989) that understanding of public 
discussions and reports about socially important topics such as health and environmental issues is 
impossible without using the language of mathematics. 

37. Eurther, it has been claimed that in a society in which the media constantly present information in 
numerical or graphical form to all citizens, the ability to interpret quantitative and statistical messages is 
vital for all adults (Paulus, 1995; Steen, 1997). It is essential for all adults to possess the ability to critically 
reflect on quantitative information encountered in various media sources and documents (Erankenstein, 
1989), and to understand how to be a careful or critical consumer of statistical arguments of various kinds 
(Gal, 2002; Utts, 2003; Watson & Callingham, 2003). 

38. Indeed, the dialogue about the various demands on adults’ knowledge has been reflected in part 
in the emphasis in PISA on the assessment of mathematical literacy and science literacy. Such constructs 
pertain, broadly speaking, to school students’ readiness for entering adults’ life contexts; it is indicative 
that they have been chosen to be the focus of assessment rather than more traditional notions of formal 
knowledge in mathematics or science areas which were assessed primarily in earlier studies. 

2.2.4 More on numeracy situations and demands. 

39. The discussion above suggests that numeracy is required so that people can effectively cope with 
or respond to a range of situations that are embedded in a life stream with real, personal meaning to them. 
The situations that call for activation of the numeracy competency can be situated in a hypothetical 
’’numeracy task space”, defined by dimensions such as the nature of the required response, the number and 
characteristics of the quantitative elements in the situation, or the extent and nature of literacy processes 
involved. Based on Gal (2000), below are described three key types of situations which illustrate the range 
of numeracy demands placed on adults. 

1 . Generative situations demand that people count, quantify, compute, or otherwise manipulate 
numbers, concrete objects, visual elements, and so forth, to create/generate new numbers or 
estimates. Examples are calculating the total price of products while shopping, finding the 
number of boxes in a crate, measuring the area of a room to be painted in order to calculate 
the amount of materials needed to do the job, reading a menu and computing the cost of a 
specified meal, filling out an order form for a product, figuring out travel times between train 
stations based on a timetable, and so forth. Generative situations include computational or 
quantitative literacy tasks (Kirsch et al, 1993), but certainly go beyond them, such as when 
measurements have to be made regarding length, volume, time, etc. 
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The numerical information in many types of generative situations may be evident in the 
situation itself (e.g., real objects to be arranged, sorted, counted, or measured; a graph on a 
computer display). Yet, numerical information may also be communicated through text or be 
embedded in different types of text; hence, such situations may also involve language s ki lls 
to varying degrees. In generative situations, tools such as a hand-held calculator, a computer- 
based application, or a measuring tape or ruler may ease the mechanics of performing needed 
calculations or increase accuracy, although a person may choose not to use them. Even when 
such tools are used, a person still needs to know how to use them efficiently and effectively, 
and his or her operations are at times more likely to be correct or accurate if the person has 
alternative strategies (e.g., mental calculations) to check on accuracy and completeness of 
actions performed. 

2. Interpretive situations demand that people make sense, and grasp the implications, of 
messages that contain information of a mathematical or statistical nature but that do not 
involve direct manipulation of numbers. An example is being faced, when reading the 
newspaper, with a report of results from a recent opinion poll based on a small sample, and 
having to decide if to take as valid a generalisation made by the writer about differences 
between two populations. Paulus (1995) describes many mathematical statements made in the 
media that call for careful consideration of their validity; other examples can be added where 
references to proportions, averages, samples, bias, correlation, risk, or causality are discussed 
or implied, such as in the context of genetic or medical counseling, or understanding of 
statistical process control displays. 

In simple interpretive situations, such as when one has to read nutrition information or a drug 
label to decide if a product contains certain ingredients (e.g., sugar, allergens, contaminants) 
whose dosage exceeds an allowed limit, the response can be assessed as correct or incorrect. 
Yet in more complex interpretive situations, the response expected is usually the creation of 
an opinion, and to create this opinion one needs to invoke and answer a set of critical before 
the information or arguments presented are accepted as credible or valid. In such cases, the 
response, i.e., opinion, has to be judged in terms of its reasonableness and the quality of the 
arguments or evidence on which it is based. Thus an opinion cannot always be easily 
classified as “right/wrong” or "accurate/inaccurate", as with responses to many generative 
situations. 

3. Decision situations demand that people locate and consider multiple pieces of information in 
order to determine a course of action, typically in the presence of conflicting goals, 
constraints, or uncertainty. Two key subtypes here are: Optimisation tasks, which require the 
identification of optimal ways to use resources such as money or supplies, or schedule 
personnel or time (see SCANS, 1991); and Choice tasks, which require a choice among 
alternatives, such as which of several apartments to rent, which pension or health insurance 
plan to join, whether to undergo a surgical medical procedure which has known probabilities 
of certain side effects. 

It is important to note that optimisation and choice tasks can be part of a broader problem- 
solving process, where alternatives have to be generated and then evaluated. Thus, what is 
being termed here a decision situation can at times also be viewed as a problem-solving 
situation. 

As with interpretive situations, a response to a decision situation has to be assessed in terms 
of its reasonableness, because the response may be based on multiple pieces of quantitative 
information (e.g., timetables, financial figures, statistical trends, event probabilities), the 
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response may be created through a process that involves both generative and interpretive 
steps, and the response is shaped after a person evaluates the quality of the final decision or 
choice against external contextual criteria. An example is when a small business owner has to 
compare financial information from several hanks to decide which loan schedule is the best 
or most manageable, given the current financial situation and anticipated costs and revenues 
of the business. The assessment of the reasonableness of a response in a decision situation 
may be further complicated, beyond what happens in an interpretive situation, because the 
response in a decision situation may also be shaped by a person’s subjective preferences and 
value system, assumptions he or she makes about future trends or event probabilities, and 
other factors. 

40. To be sure, the three types of numeracy situations described above are not mutually exclusive, 
and other cases may exist, possibly of a hybrid nature. Further, in considering the implications of these and 
other types of situations for the numeracy competency required of adults, it is important to keep in mind 
the impact of evolving technologies. As has been argued and documented by many sources (Expert Group 
on Future Skills Needs, 2007; Gatta, Appelbaum, & Boushey, 2007; Karoly, 2007), and summarised in the 
PI A AC planning documents, adults are presented with ever-increasing amounts of information of a 
quantitative nature through Internet-based or technology-based resources. More so than in prior decades, 
more types of quantitative information are more readily available, but this information has to be located, 
selected or filtered, interpreted, at times questioned and doubted, and analyzed for its relevance to the 
responses needed, whether generative, interpretative, or decision-oriented. 

2.3 Towards a definition of numeracy for PIAAC 

41. Reaching a consensus on a definition of numeracy that can fit an international programme of 
assessment is a challenging undertaking. First, as noted above there are various country-specific 
connotations for numeracy, if such a term at all exists in a local language. Second, there are overlapping or 
competing constructs such as quantitative literacy, mathematical literacy, functional mathematics, and so 
forth (Hagedorn, Newlands, Blayney, & Bowles, 2003). Third, an attempt to discuss the definition and 
meaning of numeracy is complicated by the fact different stakeholders already view it from within a given 
lens imposed by the historical and cultural aspects, whether organisational, social, economic, or linguistic, 
of the systems within which they operate. For example, some of the existing conceptions of numeracy were 
developed by educators working in delivery systems for school children, while other stakeholders link the 
term numeracy only to adult-related competencies. 

42. With the above in mind, and using the conceptions of competence and of the contexts for 
numeracy presented earlier as a backdrop, the remainder of this section is organised as follows: First, a 
review of some of the many perspectives on numeracy is presented so as to portray the key ideas that prior 
workers and scholars have addressed when discussing numeracy. Next, a definition of numeracy for 
PIAAC is presented, followed by a discussion of the facets of numerate behavior, including key content 
areas of mathematical knowledge and other cognitive and non-cognitive enabling processes and factors 
which take part in or affect numerate behavior. Finally, differences between numeracy and the constructs 
assessed in PISA and lALS are examined, in particular mathematical literacy and quantitative literacy. 

2.3.1 Perspectives on [adult] numeracy 

43. Formulation of what numeracy encompasses have evolved since the term was introduced in the 
1959 Crowther Report in England and Wales. Maguire and O’Donoghue (2003) have recently reviewed 
and organised conceptions of numeracy from several countries (Ireland, Canada, USA, UK, the 
Netherlands, Denmark and Australia) along a continuum of increasing levels of complexity or 
sophistication. Formative conceptions view numeracy as related to basic arithmetic skills. Mathematical 
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conceptions consider numeracy in a contextualised way, as a broader set of mathematical knowledge and 
skills (beyond basic computations) of relevance in everyday life. Finally, integrative conceptions consider 
numeracy as a multifaceted, sophisticated construct incorporating not only mathematics but also 
communicative, cultural, social, emotional, and personal elements which interact and pertain to how 
different people function in their social contexts. (Coben, 2000; Condelli, Safford-Ramus, Sherman, 
Coben, Gal, & Hector-Mason, 2006). 

44. At this time, formative conceptions which view numeracy as basic computational facility are 
often associated with how numeracy is viewed in connection with goals of primary schooling, and 
reflected in how numeracy is defined when classifying literacy/numeracy levels worldwide (UNESCO, 
1997). Most extant conceptions which adult education, workplace training, and national and international 
assessments have adopted fall at different points across the mathematical and integrative phases described 
by Maguire and O’Donoghue. Below are four different but related views of numeracy, the first pair from 
the UK, and the second pair from Australia. These definitions illustrate that conceptions evolve over time 
and that variability can be noticed even within the same national system. 

[numeracy is]. ..an ‘at-homeness’ with numbers and an ability to make use of mathematical skills 
which enables an individual to cope with the practical mathematical demands of his everyday 
life... [and] an ability to have some appreciation and understanding of information, which is 
presented in mathematical terms, for instance graphs, charts or tables or by reference to 
percentage increase or decrease (From the Cockcroft report: Department of Education and 
Science/Welsh Office (1982), p. 11). 

The ability to use mathematics at a level necessary to function at work and in society in 
general... understand and use mathematical information; calculate and manipulate mathematical 
information; interpret results and communicate mathematical information (From the UK 
government’s Skills for Life strategy to improve standards of adult literacy and numeracy in 
England, DfEE, 2001, p.3). 

Numeracy is the mathematics for effective functioning in one ’s group and community, and the 
capacity to use these skills to further one’s own development and of one’s community (Beazley 
committee, 1984, Australia). 

Numeracy involves abilities that include interpreting, applying and communicating mathematical 
information in commonly encountered situations to enable full, critical and effective participation 
in a wide range of life roles (Queensland Department of Education, 1994, Australia). 

45. An interesting case study of defining numeracy is offered by Lindenskov and Wedege (2001). 
Based on their work in adult and mathematics education in Denmark, they have imported numeracy from 
English-speaking countries and introduced a new term, Numeralitet, with a conceptual framework that was 
later adopted by the Danish Ministry of Education. According to this perspective, it is essential to 
distinguish between what numeracy is, or ought to be, from the individual’s and from society’s points of 
view. Lindenskov and Wedege (2001) advocate a societal view, whereby numeracy is seen as a 
competence that involves a dynamic interaction between functional mathematical skills and conceptions 
and operations on the one hand, and a series of activities and various types of data and media on the other. 
They argue that this skill- and activity-based view should be coupled with the understanding that in 
principle all people need to have this competence, and that numeracy is a competence determined by 
society and technology and that it changes in time and space along with social change and technological 
development. 



18 




EDUAVKP(2009)14 



46. Other views of numeracy, usually developed by adult education experts, focus on the role of 
adults as reflective communicators and critical consumers of information in society who are involved in the 
exchange and interpretation of messages encountered in media or in political and community contexts 
(Frankenstein, 1989). Johnston (1994) argues: 

“To be numerate is more than being able to manipulate numbers, or even being able to ‘succeed’ 
in school or university mathematics. Numeracy is a critical awareness which builds bridges 
between mathematics and the real-world, with all its diversity” (Johnston, 1994). 

47. The definition quoted from the UK’s Cockcroft Committee (1982) has been quite influential in 
that its conception of numeracy implied it is an ability to cope with various functional tasks in real-world 
contexts as well as interpretive tasks, but also pointed to the centrality of underlying supporting non- 
cognitive components. These key ideas are reflected, albeit with different terminologies and foci, in other 
views of numeracy. Another important commonality is the presence of mathematical elements or ideas in 
real situations, and the notion that these can be used or addressed by a person in a goal-oriented way, 
dependent on the needs of the individual within the given context, i.e., home, community, workplace, 
societal action, etc. 

2.3.2 Numeracy -literacy connections 

48. Several scholars and projects have pointed to the need to consider literacy when discussing 
numeracy, as the two are related and can affect each other (Baker & Street, 1994). Examples are when 
quantities are described in words and not in numbers, or appear within surrounding text whose 
interpretation is essential to understanding what is being required in terms of computations, or when there 
is a need to understand the mathematical relationships described in simple phrases, e.g., realizing that “four 
more than” is a different relationship than “four times as much.” 

49. An important aspect of the literacy-numeracy linkage has been acknowledged early on by the 
Kirsch & Mosenthal construction of literacy as comprised of Prose, Document, and Quantitative 
dimensions (Kirsch, Jungblut, & Mosenthal, 1998). Accordingly, adults skills in dealing with arithmetical 
operations embedded in text have been assessed by the Quantitative Literacy scale in lALS and several 
prior national studies. However, other areas where literacy and numeracy are linked do exist and need to be 
recognised, such as in the context of interpretation of statistical arguments in media articles (Gal, 2002a), 
or comprehending financial information which pertains to planning one’s retirement pensions or medical 
benefits. Thus, while it is possible to define numeracy in general terms without invoking literacy, as all the 
definitions quoted above have done, the structure of the tasks and demands in adults’ lives shows that these 
areas cannot be considered as mutually exclusive. Mathematical or statistical information is carried by or 
embedded in text in some, but certainly not all, contexts in which adults have to function. To the extent this 
happens, one's performance on numeracy tasks will depend not only on formal mathematical or statistical 
knowledge but possibly also on literacy-related factors such as vocabulary, reading comprehension, 
reading strategies, or prior literacy experiences. 

2.3.3 Numeracy and numerate behavior in the ALL survey. 

50. The above ideas informed the conceptualisation of numeracy for the Adult Literacy and Life 
Skills Survey (ALL), which was developed in 1998-2000 by an international team (Gal, van Groenestijn, 
Manly, Schmitt, & Tout, 2005). This was the first time the construct of numeracy had to be defined in a 
comparative assessment context and not purely in an educational context. Cognizant of the complexity and 
multi-faceted nature of the numeracy construct, the ALL team developed a three-tier conceptualisation 
which attempted to reflect key perspectives of numeracy on the one hand, but also enable 
operationalisation of the construct in an assessment scale on the other (Tout, 2006). The three tiers are a 
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brief definition of numeracy, a more elaborate definition of numerate behavior, both presented below, and 
a detailed listing of components of the facets of numerate behavior (see Gal et al., 2005). 

Numeracy is the knowledge and skills required to effectively manage and respond to the 
mathematical demands of diverse situations. 

Numerate behavior is observed when people manage a situation or solve a problem in a real 
context; it involves responding to information about mathematical ideas that may be represented 
in a range of ways; it requires the activation of a range of enabling knowledge, factors, and 
processes. 

5 1 . Both the brief and elaborate definitions shown above were seen by the ALL numeracy team to be 
required, given the needs of a comparative assessment. A brief definition is essential to simplify 
communication with various stakeholders, such as policy-makers and experts. However, as with most brief 
definitions of complex constructs, the language used is broad and abstract, hence the definition cannot be 
explicit about what a numerate person can do and what behavior to observe in an assessment. With this in 
mind, the more detailed definition of numerate behavior was developed as a way to emphasise four key 
facets or dimensions which were seen by the ALL numeracy team as underlying numerate behavior, as 
follows: 



• Contexts'. The range of external demands (e.g., work, home, etc.); 

• Responses'. What a person can do in response to the external demands (e.g., compute, interpret, 
communicate, etc.); 

• Mathematical ideas/content'. The informational content which the context carries or enables 
access to, which can be seen as mathematical (or statistical) in nature and hence of interest in the 
context of an assessment of numeracy (e.g., numbers, proportions, measurements, statistical 
concepts, etc); 

• Representations'. The different ways in which the mathematical (or statistical) information exists 
or is conveyed to the person in the given context (e.g., text, numbers, graphs, etc.) 

52. The advantage of using a more elaborate definition of numerate behavior was that it is more 
explicit about what to examine in an assessment, and thus serves as a springboard for developing an actual 
specification for an assessment scale. It is important to also note that the definition of numerate behavior 
points to the presence of both cognitive and non-cognitive factors which underlie or enable effective 
numerate behavior. Ideally, coverage of both cognitive and non-cognitive aspects of numerate behavior is 
essential in order to generate a full picture regarding the competence of numeracy. 

2.3.4 Definition of numeracy for PIAAC. 

53. The development of the conceptualisation and definition of numeracy for PIAAC went through 
several stages of work and consultation. An expert panel appointed to develop the overall assessment 
design for PIAAC presented in summer 2006 tentative recommendations regarding all competencies to be 
assessed in PIAAC (OLCD, 2006) and then proposed to define numeracy as: "The ability to use, apply, and 
communicate mathematical information". Various perspectives on numeracy and its assessment were later 
examined by participants at the Canada-OLCD Expert Technical Workshop on Numeracy, which met in 
November 2006 in Ottawa; a tentative working definition of numeracy was then proposed for PIAAC and 
included in a draft framework circulated for external review (Gal, 2007). Lurther development of the 
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numeracy framework has been undertaken by the Numeracy Expert Group for PIAAC appointed in April 
2008, which released a revised framework for review by all participating countries in October 2008. 

54. In general, work on the development of a numeracy framework for PIAAC, together with the 
assessment scale and related item pool, has been conducted with two somewhat conflicting objectives in 
mind. One objective is the need to maintain compatibility with the conceptualisation of numeracy in the 
ALL survey, given the need for PIAAC to provide trend data related to ALL results. Eor this reason 
PIAAC was designed with a specification that 60% of the literacy and numeracy tasks that will be 
employed in the final assessment scale will come from item pools used in ALL and lALS. The other 
objective is the need to extend the ALL definition in light of PIAAC's overarching conceptualisation of 
“literacy competencies in the information age", and consider new or emerging uses of numeracy in the 
adult world. 

55. Taking all the above into consideration, numeracy has been defined for PIAAC as follows: 



Numeracy is the ability to access, use, interpret, and communicate mathematical 
information and ideas, in order to engage in and manage the mathematical demands of a 
range of situations in adult life 



56. This definition captures essential elements in numerous conceptualisations of numeracy in the 
extant literature; it is compatible with the definition used for ALL and appears to provide a solid basis from 
which to develop an assessment scale for PIAAC with its emphasis on competencies in the information 
age. The inclusion of "engage" in the definition signals that not only cognitive skills but also dispositional 
elements, i.e., beliefs and attitudes, are necessary for effective and active coping with numeracy situations. 
It is also important to note that while the definition of numeracy for PIAAC has been developed in the 
context of an assessment programme, it has been crafted so as to contribute to public dialogue regarding 
the goal of educational and social interventions focused on developing adult competencies in general, and 
adult numeracy and related mathematical and statistical skills and dispositions in particular. 

57. However, since numeracy is a broad, multifaceted construct referring to a complex competency, 
the definition of numeracy given above should not be considered by itself, but should be coupled with a 
more detailed definition of numerate behavior and with further specification of the facets of numerate 
behavior. This pairing is essential in order to enable operationalisation of the construct of numeracy in an 
actual assessment, thereby contributing to the assessment’s validity and interpretability, and in order to 
further broaden the understanding of key terms appearing in the definition itself. Consequently, a definition 
of numerate behavior similar in general terms to the one used for the ALL survey, but shorter, has been 
adopted for PIAAC: 



Numerate Behavior involves managing a situation or solving a problem in a real context, 
by responding to mathematical content/information/ideas represented in multiple ways. 



Table 1 : Numerate behavior - key facets and their components 



Numerate behavior involves managing a situation or solving a problem... 

1 . in a real context: 

- everyday life 

- work 

- societal 

- further learning 

2. by responding: 

- identify, locate or access 
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- act upon, use: order, count, estimate, compute, measure, model 

- interpret 

- evaluate / analyze 

- communicate 

3. to mathematical content/ information/ ideas: 

- quantity & number 

- dimension & shape 

- pattern, relationships, change 

- data & chance 

4. represented in multiple ways : 

- objects & pictures 

- numbers & mathematical symbols 

- formulae 

- diagrams & maps, graphs, tables 

- texts 

- technology-based displays 



Numerate behavior is founded on the activation of several enabling factors and processes: 

- mathematical knowledge and conceptual understanding 

- adaptive reasoning and mathematical problem-solving skills 

- literacy skills 

- beliefs & attitudes 

- numeracy-related practices and experience 

- context/world knowledge 



58. The definition of numerate behavior pertains to four facets: Contexts, Responses, Mathematical 
content/information/ideas. Representations. Table 1 lists the components of the four facets, and these 
components are explained in more detail the next section. Table 1 is based on the original description of 
the facets of numerate behavior developed for the ALL survey, but some changes have been implemented, 
such as the addition of "access" and of "evaluate/analyze" as possible responses, the merging of the content 
categories of "change" and "pattern and relationship", or the reference to "technology-based displays" as 
another representation mode. 

59. It should be noted that the bottom part of Table 1 also lists several enabling factors and processes 
elaborated in section 3.5, whose activation underlies numerate behavior. Most of these enabling factors and 
processes appeared in the ALL conceptual framework, but some changes were introduced, such as the 
positioning of "adaptive reasoning and mathematical problem-solving" as a separate enabling factor. 
Overall, the definition of numerate behavior presented earlier, together with the details in Table 1 and the 
further explanations below, provided a roadmap for the development of a numeracy scale for PIAAC. 
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PART 3. FACETS OF NUMERACY 



3.1 Facets of numerate behavior 

60. This section^ elaborates on the facets of numerate behavior and their components, listed in Table 
1. The discussion of the first facet, context, mainly revisits ideas which appeared in section 2.2 above 
regarding contexts and demands for numeracy. Elaborations on the other facets are based on materials and 
ideas in the bibliographic sources mentioned earlier, and the analysis of the components of adult numeracy 
by Ginsburg et al. (2006) which was based on an integrative review of multiple numeracy frameworks 
from several countries (see also Hagedom et al, 2003). This section has also benefited from the positions 
presented in a report of the UK’s National Research and Development Centre for Adult Literacy and 
Numeracy (NRDC, 2006), background papers prepared for the OECD-Canada Expert workshop on 
numeracy (Nov. 2006, Ottawa) and suggestions made by workshop participants, external reviews of earlier 
drafts of this framework, and professional perspectives of PIAAC's Numeracy Expert Group. 



3.1.1 Facet 1: Contexts. 

61. People try to manage or respond to a numeracy situation because they want to satisfy a purpose 
or reach a goal. Pour types of contexts where demands on people’s numeracy may appear are described 
below. These are not mutually exclusive and may involve the same underlying mathematical themes. 

a. Everyday life. The numeracy tasks that occur in everyday situations are often 
encountered in personal and family life, or revolve around hobbies, personal 
development, and interests. Representative tasks are handling money and budgets, 
comparison shopping, personal time management, making decisions involving travel, 
planning holidays, mathematics involved in hobbies like quilting or wood-working, 
playing games of chance, understanding sports scoring and statistics, reading maps, and 
using measurements in home situations such as cooking or home repairs. 

b. Work-related. At work, one is confronted with quantitative situations that often are 
more specialised than those seen in everyday life. In this context, people may develop 
good skills in managing situations that might be narrower in their application of 
mathematical themes. Representative tasks are completing purchase orders, totaling 
receipts, calculating change, managing schedules, budgets, and project resources, using 
spreadsheets, organizing and packing different shaped goods, completing and 
interpreting control charts, making and recording measurements, reading blueprints, 
tracking expenditures, predicting costs, and applying formulas. 

c. Societal or community. Adults need to know about trends and processes happening in 
the world around them (e.g., regarding crime, health issues, wages, pollution) and may 
have to take part in social events or community action. This requires that adults can 



3 Selected text portions in section 3.1, as well as in subsections 3.1.3 - 3.1.5, were adapted and expanded 

from the numeracy framework developed in 1998-2000 for the ALL survey, and used here with permission 
of Statistics Canada. The final version of the ALL numeracy framework is presented by Gal, van 
Groenestijn, Manly, Schmitt, & Tout (2005). 
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read and interpret quantitative information presented in the media, including statistical 
messages and graphs. Also, they may have to manage situations like organizing a fund- 
raiser, realizing the fiscal effect of community programs, or interpreting the results of a 
study about a health issue. 

d. Further learning. A numeracy competency may enable a person to participate in 
further study, whether for academic purposes or as part of vocational training. In either 
case, it is important to be able to know some of the more formal aspects of mathematics 
that involve symbols, rules, and formulas and to understand some of the conventions 
used to apply mathematical rules and principles. 

62. It should be emphasised that performance in all of the above contexts is based on a combination 
of cognitive and non-cognitive elements, and thus requires that we think of numeracy as a competence as 
defined earlier, not just as possession of a set of technical skills or know-how. For example, engagement in 
further learning of mathematical topics, whether in formal or informal contexts, involves willingness to 
start such learning in the first place as well as perseverance in such learning. For such engagement to 
occur, an adult is required to have positive beliefs and attitudes about mathematics and about oneself as a 
person capable to cope with mathematical tasks. 

3.1.2 Facet 2: Responses. 

63. In different types of real-life situations, people may have to react with diverse types of responses, 
grouped below under three broad headings: Identify, locate, or access; Act upon or use; and Interpret, 
evaluate/analyse, communicate. It should be noted that while these types of responses are described 
separately, in real life they may co-occur in a dynamic fashion, and may vary from simple to more 
complex. The extent to which response types will vary in complexity or co-occur in an integrated way 
depends on various aspects of the situation at hand discussed later, such as the density of the information 
available, presence of distracting information, transparency of the task, literacy demands, number of steps 
and iterations involved. Further, responses are shaped by the interaction between situational demands on 
the one hand and the goals, skills, dispositions, and prior practices and experiences of the person on the 
other hand. 

a. Identify, locate, or access. In virtually all situations, people have to identify, locate or 

access some mathematical information present in the task or situation confronting them 
that is relevant to their purpose or goal. When it exists alone, this response type often 
requires only low level mathematical understanding or application of simple arithmetic 
skills. Usually, however, this response type is subsumed or co-occures with the other 
types of responses listed below. 

b. Act upon or use. In situations described earlier (see 3.2.4) as “generative”, people have 

to perform actions on the mathematical information which can be identified in the 
situation, or use known mathematical procedures and rules. Acting upon or using 
encompasses arithmetical operations such as when counting, doing calculations “in the 
head”, with pen and paper or with a calculator. Acting upon or using may also involve 
ordering or sorting, estimating, figuring out an area or volume of a certain object in an 
approximate way, or using various measuring devices to generate needed mathematical 
information of a more exact nature. Finally, acting upon may involve using (or 
developing) a formula which serves as a model of a situation or a process. 

c. Interpret, evaluate/analyse, communicate. This response type encompasses three 
separate but related responses, described below in more detail: 
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• Interpret. Some situations do not demand any direct manipulation or action on 

available quantitative information, but tbe interpretation of tbe meaning and 
implications of given information of a mathematical or statistical nature. Further, in 
such situations, described earlier as “interpretive”, the person in the situation may 
need to not only interpret mathematical or statistical information but also make a 
judgment or create an opinion, such as about trends, changes, or differences 
described in a graph or in a text appearing in a newspaper article or advertisement. It 
should be emphasised that interpretive responses may be in reaction not to 
information that is numerical (i.e., figures or statistical data), but to broader 
mathematical or statistical concepts possibly expressed in oral, textual, or visual 
manner, including ideas such as rate of change, proportions, shape of distribution, 
samples, bias, correlation, probability or risk, or causality. 

• Evaluate/analyse. This response category is in part an extension of the Interpret 
response type described above, which was used in ALL. It accommodates responses 
that may be more likely in situations requiring a person to analyze a problem and in 
so doing evaluate the quality of the solution against some criteria or contextual 
demands, and if needed cycle again through the interpretation, analysis and 
evaluation stages. Such situations may be encountered in various contexts, including 
in dynamic or information-rich technology environments, or those termed earlier 
'decision situations'. Examples are when, as part of developing a solution to 
demands of a given situation, an adult has to process raw quantitative information 
through technology-enabled channels (e.g., sift through a website of a national 
statistical office), or retrieve and integrate information from multiple sources after 
evaluating their relevance to the task at hand (e.g., compare information from 
different sources regarding costs of competing courses of action). 

• A common feature that pertains both to "interpret" and "evaluate/analyze" responses 
is that the judgments or opinions expected in the situation may need to be critical in 
nature. Examples are when a person needs to question the validity of the data or 
information presented, identify gaps between the available information and the 
conclusions presented by a source (e.g., a journalist or politician), or reflect about 
the proposed implications of the data, both for himself or herself as an individual or 
for the wider community. As a result, these two responses types have an inherent 
overlap between them. 

• Communicate. In addition to the responses listed above, a person may have to 
represent and communicate about the mathematical information given, describe the 
results of one’s actions or interpretations to someone else, or explain and justify the 
logic of one’s analysis or evaluation. This can be done via oral or written means 
(ranging from presenting a simple number, a word, all the way to a detailed 
explanation), through a drawing (a diagram, map, graph) or by generating a 
computer-based display (e.g., by referencing a spreadsheet-based chart showing the 
results of “what if’ scenarios), and various combinations of these and other modes 
of communication and illustration. 

64. Problem-solving is not seen as a separate response type, but rather assumed to be part of the 
demands set forth by the external situation - as is implied in Table 1, the goal of numerate behavior is 
managing a situation involving a numeracy task or solving a numeracy-related problem. Hence, several 
response types described above may be called upon and co-occur when people have to solve numeracy- 
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related problems, especially novel ones. Such responses may be aided or organised by more generalised 
skills of adaptive reasoning and problem-solving, examined later on as part of "enabling processes" that 
underlie numerate behavior. 

Note about assessment of certain response types: 

65. The ideas above describe key ways in which people may respond to mathematical/statistical tasks 
embedded in a range of real-life situations. However, one needs to distinguish between a conceptual 
framework (discussed in this section) and an assessment framework (discussed later on). Not all real life 
numeracy tasks can necessarily be simulated well in a specific assessment. Further, the ability of an 
assessment to actually capture, evaluate, and score responses associated with numerate behavior 
ultimately depends on the technical aspects of that assessment. While the computer-based assessment 
platform chosen for PIAAC offers many advantages, the design specifications adopted by participating 
countries pose limitations on skills assessments in PIAAC, due to the computer-based environment 
employed for the assessment, the need for immediate scoring of responses given the adaptive testing 
process necessary for efficient ability estimation, and restrictions on testing time per respondent which are 
typical in large scale household surveys. These realities necessitate the use of short separate tasks, exclude 
extended problem tasks, and prohibit the use of most types of numeracy tasks that require respondents to 
communicate via free-form text input. Specifically, tasks requiring communication-based responses, such 
as when adults have to explain interpretations of given information, or describe their evaluation or analysis 
of a situation or their thinking about that situation, could hardly be used in the direct assessment of all 
skills targeted by PIAAC. Such tasks do comprise an important, inseparable part of the landscape of adult 
numeracy situations and are an inherent part of the conceptual framework of adult numeracy, yet very few 
could be included in the item pool for the first cycle of PIAAC. 

3.1.3 Facet 3: Mathematical content/information/ideas. 

66. Mathematical information can be classified in several ways and on different levels of abstraction. 
One approach is to refer to fundamental “big ideas” in mathematics. Steen (1990), for example, identified 
six broad categories: Quantity, Dimension, Pattern, Shape, Uncertainty, and Change. Rutherford & 
Ahlgren (1990) described networks of related ideas: Numbers, Shapes, Uncertainty, Summarizing data. 
Sampling, and Reasoning. Dossey (1997) categorised the mathematical behaviors of quantitative literacy 
as: Data representation and interpretation. Number and operation sense. Measurement, Variables and 
relations. Geometric shapes and spatial visualisation, and Chance. More broadly, many curriculum 
frameworks around the world in one way or another refer to these key areas, albeit using somewhat 
different terminologies and with somewhat different groupings (e.g., NCTM, 2000). 

67. Based on such and related classifications, the ALL numeracy framework (Gal et al., 2005) 
defined five areas of mathematical content and ideas that characterise the mathematical demands faced by 
adults: Quantity & number; Dimension & shape; Data & chance; Pattern, functions, & relationships; 
Change. In PIAAC the first three areas were retained, while the last two areas listed for ALL were united 
to create a single area called “Patterns, relationships and change”. Such a change was deemed sensible 
because these two areas are sufficiently related. Indeed, Ginsburg et al, 2006, in their analysis of the 
components of numeracy, subsumed “change” as part of a unified domain termed “patterns, functions, and 
algebra.” Uniting the two areas has to advantages: it enables PIAAC to focus on domains that are more 
distinct from each other, as well as maintains better conceptual compatibility with PISA, an issue discussed 
in Section 5. 

68. With the above in mind, four key areas of mathematical content, information and ideas are 
covered by the numeracy assessment in PIAAC and are briefly summarised below. 
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a. Quantity and Number. Quantity is described by Fey (1990) as an outgrowth of 
people’s need to quantify the world around us, using attributes such as: numbers of 
features or items; costs and charges for goods and services; size (e.g. length, area, and 
volume); temperature, humidity, and pressure of our atmosphere; populations and 
growth rates of species; revenues or profits of companies, etc. Number is fundamental 
to quantification and different types of number constrain quantification in various ways: 
whole numbers can serve as counters or estimators; fractions, decimals and percents as 
expressions of greater precision, parts or comparisons; and positive and negative 
numbers as directional indicators. In addition to quantification, numbers are used to put 
things in order and as identifiers (e.g., telephone numbers or zip codes). There is also 
the requirement to operate on such quantities and numbers (the four main operations of 
+, -, X, and others such as squaring). Facility with quantity, number, and operation on 
number requires a good "sense" of magnitude. Contextual judgment comes into play 
when deciding how precise one should be or which tool (calculator, mental math, a 
computer) to use. Money and time management, the ubiquitous mathematics that is part 
of every adult's life, depends on a good sense of number and quantity. A basic level 
numeracy task might be figuring out the cost of one can of soup, given the cost of 4 for 
$2.00; a task with a higher cognitive demand could involve more complex numbers 
such as when figuring out the cost when buying 0.283 kg of cheese at 12.95 Furos per 
kg- 

b. Dimension and shape. Dimension includes “big ideas” related to one, two, and three 
dimensions of “things” (using spatial and numerical descriptions), projections, lengths, 
perimeters, areas, planes, surfaces, location, etc. Facility with each dimension requires a 
sense of "benchmarks" and estimation, direct measurement and derived measurement 
skills. Shape is a category describing real images and entities that can be visualised 
(e.g., houses and buildings, designs in art and craft, safety signs, packaging, 
snowflakes, knots, crystals, shadows and plants),in both two and three dimensions . 
Direction and location are fundamental qualities called upon when reading, interpreting 
or sketching maps and diagrams. This content area requires an understanding of units 
and systems of measurement, both informal and standardised such as the Metric and 
Imperial systems. A basic numeracy task in this fundamental aspect could be shape 
identification whereas a complex task might involve describing the change in the 
capacity of an object when one dimension is changed. 

c. Pattern, relationships, and change. It is frequently written that mathematics is the 
study of patterns and relationships. Pattern is seen as a wide-ranging concept that 
covers patterns encountered all around us, such as those in musical forms, nature, traffic 
patterns, etc. It is argued by Senechal (1990) that our ability to recognise, interpret, and 
create patterns is the key to dealing with the world around us. The human capacity for 
analyzing and identifying patterns and relationships undergirds much mathematical 
thinking. Relationships and change relate to the mathematics of how things in the world 
are associated or develop. Individual organisms grow, populations vary over time, 
prices fluctuate, and objects traveling speed up and slow down. Some characteristics or 
values can change directly in proportion or relation to another change, whilst other 
characteristics may change in the opposite direction or in a different way. Change and 
rates of change help provide a narration of the world as time marches on. The ability to 
generalise and to characterise relationships between variables is a crucial gateway to 
understanding basic economic, political or social analyses. This domain includes the 
ability to develop and/or use a mathematical formula between the different variables 
involved in a situation, alongside the need to be able to understand, use and apply a 
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sense of proportional reasoning. A lower level numeracy task may ask someone to use a 
familiar formula such as that for calculating the area of a square or rectangle. More 
demanding tasks involving relationships and change may require using formulae such 
as for calculating compound interest or one’s BMI (Body Mass Index). Or, tasks could 
require using an electronic spreadsheet or a Weh-hased dedicated calculator (applet) for 
exploring “what if’ scenarios related to different interest rates, or to different levels of 
weight loss or weight gain, and their impact on one’s long-range savings or health risk 
levels, respectively. 

d. Data and chance. Data and chance encompass two related hut separate topics. Data 
covers “hig ideas” such as variability, sampling, error, or prediction, and related 
statistical topics such as data collection, data displays, and graphs. Modern society 
demands that adults interpret and produce organisers of data such as frequency tables, 
pie charts, graphs and to sort out relevant from irrelevant data. Chance covers “big 
ideas” related to probability, subjective probability, and relevant statistical methods. 

Few things in the world are 100% certain; thus the ability to attach a number that 
represents the likelihood of an instance is a valuable tool whether it has to do with the 
weather, the stock-market, or the decision to board a plane. In this mathematical 
category, a simple numeracy skill might be the interpretation of a simple pie chart; a 
more complex task would be to infer the likelihood of an occurrence, such as predicting 
the weather, based upon past information. 

3.1.4 Facet 4: Representations of mathematical information. 

69. Mathematical information in a situation may be available or represented in many forms. It may 
appear as concrete objects to be counted (e.g., people, buildings, cars, etc.) or as pictures of such things. It 
may be conveyed through symbolic notation (e.g., numerals, letters, and operation or relationship signs). 
Sometimes, mathematical information will be conveyed by formulae, which are a model of relationships 
between entities or variables. Mathematical information may be encoded in visual displays such as a 
diagram or chart; graphs and tables may be used to display aggregate statistical or quantitative information (by 
displaying objects, counting data, etc.). Similarly, a map of a real entity (e.g., of a city or a project plan) may 
contain information that can be quantified or mathematised. Last but not least, textual elements may carry 
much mathematical information or affect the interpretation of mathematical (and statistical) information, as 
explained further below. 

70. A person may have to extract mathematical information from various types of texts, either in prose or 
in documents with specific formats (such as in tax forms). Two different kinds of text may be encountered in 
numeracy tasks. The first involves mathematical information represented in textual form, i.e., with words 
or phrases that carry mathematical meaning. Examples are the use of number words (e.g., “five” instead of 
“5), basic mathematical terms (e.g., fraction, multiplication, percent, average, proportion), or more 
complex phrases (e.g., “crime rate increased by half’) which require interpretation, or coping with double 
meanings (or with differences in mathematical and everyday meanings of the same terms). The second 
involves cases where mathematical information is expressed in regular notations or symbols (e.g., 
numbers, plus or minus signs, symbols for units of measure, etc.), but is surrounded by text that despite its 
non-mathematical nature also has to be interpreted in order to provide additional information and context. 
An example is a bank deposit slip with some text and instructions in which numbers describing monetary 
amounts are embedded, or a parking ticket specifying an amount of money that has to be paid by a certain 
date due to a parking violation, but also explaining penalties and further legal steps that will be enacted if 
the fine is not paid by a certain date. 
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3.2 Enabling processes: cognitive and non-cognitive 

71. People's numeracy competence is revealed through their responses (i.e., identifying, interpreting, 
acting upon, evaluating, and communicating) to the mathematical information or ideas that may be 
represented in a situation or that can be applied to the situation at hand. It is clear that numerate behavior 
will involve an attempt to engage with a task and not delegate it to others or deal with it by intentionally 
ignoring its mathematical content. Numerate behavior, however, depends not only on cognitive skills or 
knowledge bases, but also on several enabling factors and processes listed in Table 1 (NRDC, 2006; Tout, 
2006). 

72. Specifically, the enabling processes involve integration of mathematical knowledge and 
conceptual understanding with broader reasoning, problem-solving skills, and literacy skills. Further, 
numerate behavior and autonomous engagement with numeracy tasks depend on the dispositions (beliefs, 
attitudes, habits of minds, etc), and prior experiences and practices that an adult brings to each situation. 
These are briefly discussed below. Most of these enabling factors and processes have also been described 
by Kilpatrick (2001) as part of his analysis of the construct of mathematical literacy, and further examined 
and deemed relevance for description of adult numeracy in a recent analysis by Ginsburg et al. (2006). 

3.2.1 Mathematical knowledge and conceptual understanding. 

73. The notion of conceptual understanding refers to an integrated and functional grasp of 
mathematical ideas (Kilpatrick, Swafford & Findell, 2001: 118). Ginsburg et al (2006) suggest that the two 
aspects of conceptual understanding, i.e., it being integrated and functional, frame the ability to think and 
act numerately and effectively, and that across different numeracy frameworks in different countries, 
equivalent terms are used such as “meaning making,” “relationships,” “model,” and “understanding.” 
Conceptual understanding can help learners produce reasonable estimates that can help them catch 
computational errors, or realise that an exact product is not necessary, but an estimate is enough for the 
purpose. Ginsburg et al (2006) further explain that conceptual understanding permits one to be free from 
relying on memory for all methods and procedures, i.e., an adult can think about the meaning of the task 
and “construct or reconstruct” a representation that both illustrates what it means and suggests a method 
for solution. As an example they state that a fundamental conceptual understandings include interpreting 
and visualizing 23 x 13 as the repeated addition of 13 objects, 23 times (one could arrive at an accurate 
answer by adding groups), or as a 23 by 13 rectangular array (one could count the elements in the array). 

3.2.2 Adaptive reasoning and problem-solving skills. 

74. Throughout life, adults develop or apply diverse strategies to manage their quantitative situations. 
Some strategies may be based on prior formal learning, while others may be self-invented or adapted to fit 
the situation at hand. To solve computational problems or to manage certain quantitative tasks, people have 
to re-construct reality in a mathematical way, for example, model or mathematise. They can do so either on 
their own or in discussion with other people. Problem-solving strategies may include, e.g., extracting 
relevant information from the task/activity; rewriting/restating the task; drawing pictures, diagrams or 
sketches; guessing and checking; making a table; and/or generating a concrete model or representation 
(Kilpatrick, 2001; Ginsburg et al., 2006). 

3.2.3 Literacy skills. 

75. The ability to read, write, and talk are important skills in undertaking a numeracy task or activity 
or communicating the outcomes of working on such tasks. In cases where “mathematical representations” 
involve text, one's performance on numeracy tasks will depend not only on formal mathematical or 
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statistical knowledge but also on reading comprehension and literacy skills, reading strategies, and prior 
literacy experiences. For example, following a computational procedure described in text (such as the 
instructions for computing shipping charges or adding taxes on an order form) may require special reading 
strategies, as text is very concise and structured. Likewise, analyzing the mathematical relationships 
described in words requires specific interpretive skills, as in the simple case of recognizing the similarity of 
“the price doubled” and “the priced was twice as high”, but the different meanings in “production levels 
were constant over the last five years” and “production levels constantly increased over the last five years”. 

3.2.4 Context/world knowledge. 

76. Proper interpretation of mathematical information or quantitative messages by adults depends on 
their ability to place messages in a context and access their world knowledge, as well as rely on their 
personal experiences and practices, noted further below. World knowledge also supports general literacy 
processes and is critical to enable “sense-making” of any message. For example, adults’ ability to make 
sense of statistical claims or media-based graphs will depend on information they can glean from the 
message about the background of the study or data being discussed. When interpreting statistical claims 
made by journalists, advertisers and the like, context knowledge is the main determinant of the reader’s 
familiarity with sources for variation and error, helps to imagine why a difference between groups can 
occur (as in a medical or educational experiment), or what alternative interpretations may exist for reported 
findings about an association or correlation between certain variables. Likewise, world knowledge is a 
prerequisite for enabling critical reflection about statistical messages and for understanding the 
implications of the reported findings. 

3.2.5 Beliefs and attitudes. 

77. Research literature suggests that the ways in which a person responds to a numeracy task, 
including overt actions as well as internal thought processes and the adoption of a critical stance, depend 
not only on knowledge and skills but also on negative attitudes towards mathematics, beliefs about one's 
mathematical skills, habits of mind, and prior experiences involving tasks with mathematical content 
(Lave, 1988; Schliemann & Acioly, 1989; Saxe, 1991). In some cultures, some adults, including highly 
educated ones, decide that they are not “good with numbers” or have other sentiments or self-perceptions 
usually attributed to negative prior experiences they have had as pupils of mathematics (Tobias, 1993). 

Such attitudes and beliefs stand in contrast to the desired sense of “at-homeness with numbers” (Cockcroft, 
1982) and can interfere with one’s motivation to develop new mathematical skills or to tackle math-related 
tasks, and may also affect test performance (McLeod, 1992). 

78. In real-world contexts, adults with a negative mathematical self-concept may elect to avoid a 
problem with quantitative elements, address only a portion of it, or prefer to delegate a problem, e.g., by 
asking a family member or a salesperson for help. Such decisions or actions can serve to reduce both 
mental and emotional load (Gal, 2000). Yet, such actions may fall short of autonomous engagement with 
the mathematical demands of real-world tasks, carrying negative consequences, e.g., not being able to fully 
achieve one’s goals. 

3.2.6 Numeracy -related practices and experiences. 

79. Research suggests that, for adults as well as for children, mathematical knowledge develops both 
in and out of school (e.g., Schliemann & Acioly, 1989; Saxe, 1991; Lave, 1998). Saxe and his colleagues 
have written about the importance of cultural practice in the development of mathematical thinking and 
how such practices profoundly influence an individual’s cognitive constructions and mathematical ideas, 
depending, e.g., on the artifacts or tools they use, the nature of the measurement systems in their culture, 
the counting or calculating devices (abacus, calculator) they use, the distribution of work among family 
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members, or general patterns and types of social activity. Further, the frequency of engaging with 
mathematical tasks or of exposure to mathematical or statistical information or displays, whether at work, 
home, when shopping, or in other contexts, is of much interest. Engagements or practices in this regard can 
be both the result of a certain skill level, but also the cause of observed skill levels, or at a minimum a 
factor influencing observed skill level apart from prior formal schooling. 

80. The ideas above suggest that numerate behavior does not rely only on mathematical knowledge 
or related reasoning and problem-solving skills acquired as part of formal learning in a school context. 

Both attitudes and beliefs as well as numeracy-related practices and world knowledge are important 
enabling processes and may influence adults' ability to act in a numerate way. Therefore, scales assessing 
selected attitudes and beliefs about mathematics, and numeracy-related practices in work, everyday, and 
other settings, have been developed for PIAAC's Background Questionnaire (BQ). Information collected 
by such scales can help to explain differences in performance among adults, further inform our 
understanding of factors that affect skill acquisition and retention or motivation for further learning, and 
explain the links between numeracy and covariates such as participation in further learning or 
employment/unemployment status. 



PART 4. SCALE DEVELOPMENT: PRINCIPLES, CONSTRAINTS, IMPLEMENTATION 



8 1 . The operationalisation of the construct of numeracy in a large-scale assessment scale is affected 
by many factors which shape the extent to which the theoretical construct can be fully addressed by the 
actual collection of items used in the direct assessment. This part first describes general expectations in 
assessing adult numeracy gleaned from prior work on assessing adults’ mathematical skills and the 
theoretical foundations reviewed above, followed by an outline of design constraints that affect the 
development of a scale for direct assessment of cognitive skills in PIAAC. Based on these foundations, an 
outline is presented of design principles that guide the assessment of numeracy in PIAAC, and further 
details on a supporting scheme regarding factors that affect task complexity (or item difficulty) which is of 
importance both for task design as well as interpretation of results regarding numeracy in PIAAC. 

4.1 General ideas about shaping tasks for assessing adult numeracy 

4.1.1 Task authenticity and realism. 

82. Numerous authors have highlighted the need to retain in assessments of adults’ numeracy the 
authenticity of assessment tasks and make them as similar as possible to the way adults encounter 
mathematics in different life contexts. Ginsburg et al (2006: 9), for example, claim: 

“If one accepts the premise that ‘realistic’ is not the same as ‘real’, a serious question is 
raised about the extent to which ‘efficient’, short-response standardised test items are 
valid measures of a person ’s numeracy when the items are not structured to elicit the 
practices an adult actually employs in a real situation 

83. It follows that differential performances can occur when assessment is divorced from, as opposed 
to contextualised in, realistic settings (Lave, Murtagh, & de la Rocha, 1984). Problem-solving in 
contextualised real life and work activities may differ from solving school-like problems (Resnick, 1987; 



31 




EDU/WKP(2009)14 



Greeno, 2003). Thus, assessments of adult numeracy have to aim for a high degree of realism and 
authenticity in both stimuli and tasks presented to respondents. The desire to retain authenticity, however, 
may at times he at odds with the need to establish cultural appropriateness of tasks and stimuli and reduce 
context effects. Tasks deemed as authentic and valid in the context of one country or culture may be 
unfamiliar to a smaller or larger degree in another cultural context. This is a traditional problem in cross- 
cultural testing that has challenged generations of test specialists. 

84. Arguably the problem of authenticity and cultural appropriateness is lessened when testing pupils 
in schools, such as in PISA, because test designers can use conventional mathematical terminology, 
formulae, symbols, and so forth; this helps school-age assessments to standardise the demands from 
respondents by conveying the mathematical information embodied in different situations in consistent 
ways regardless of the cultural context. However, testing of adults’ numeracy presents more challenges 
because many will not remember formal school-based notations or terminology. In countries where a 
sizable proportion of the population are immigrants or speak multiple home-based languages, the gaps 
between mother tongues and school-based mathematical linguistic conventions may further affect 
performance on some numeracy tasks. Thus, attention has to be given to linguistic and cultural factors 
when adapting items for adult assessments. 

4.1.2 Task format and coding. 

85. Another aspect of importance in designing assessment of adult numeracy is task format, i.e., 
forced-choice (or multiple choice) format versus a constructed-response format where respondents 
communicate in their own words the answers to tasks or questions given as part of the direct assessment, or 
otherwise are free to choose how to respond and are not limited to a specific and small set of given 
responses as in multiple -choice tests. Some of the key arguments for using constructed-response formats in 
adult numeracy assessment are that in most real-life situations, adults have flexibility in how they choose 
to respond to given tasks. Many real life tasks call for approximate answers or estimates rather than for 
accurate results, or for opinions or judgments that adults have to express in their own words. Further, there 
is long-standing awareness regarding the limited ability of forced-choice items to reflect reasoning or 
problem-solving processes and arguments that underlie the choice of a particular response. Thus, the use of 
multiple -choice items undermines the ability to assess the extent to which adults can “communicate 
mathematically”. On the other hand, the coding of the constructed responses as being correct or incorrect 
can be more complicated and require much further training of coders, while items where respondents have 
to choose from among a limited set of possible answers sometimes (but certainly not always) offer 
advantages in terms of cost, speed, and reliability of coding. 

4.1.3 Usage of calculators and other tools or objects. 

86. The assessment of numeracy, whether by paper and pencil tasks or their computer-based 
equivalents, has to take into account that the practice of numeracy in everyday or work situations also 
involves the use of certain objects and artifacts. First we should examine the use of hand-held calculators, 
which by now are inexpensive and would be widely available to adults from all walks of life in many 
countries. Calculators have been included for quite some years now in school curricula so should be 
familiar to many adults. Thus, hand-held calculators are tools which are part of the fabric of numeracy life 
in many cultures. Increasingly, respondents in large scale tests are allowed, sometimes even expected, to 
use calculators. However, we still see discussions as to whether an assessment should be conducted with, 
or without, allowing respondents to use calculators or other technological tools. 

87. It follows that adults should be given access to a calculator as part of an assessment of numeracy 
skills, and they can then choose if and how to use it. That said, when a calculator is made available, it is 
not possible to know what exactly the respondent does with it or for what it is being used in each task, e.g.. 
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does the respondent use it to compute a result which is then given as an answer to the task, or use it to 
verify results that were first obtained hy mental or manual (written) calculation? It is also difficult to 
document problems in using a hand-held calculator without having an examiner looking all the time “over 
the shoulder” of the respondent and intruding into the respondents' work process. Thus, while making a 
calculator available during testing is paramount, collecting information about its usage presents many 
challenges. Yet, without information about the purpose of usage, it is difficult to analyze whether the usage 
of a calculator helps adults cope with certain numeracy tasks, or to conclude what might be the educational 
or policy implications. 

88. In addition to a calculator, other tools or objects could be used in certain assessment tasks. The 
use of a ruler or measuring tape, whether in a metric or imperial (inches) system are part of contexts where 
adult numeracy competence is manifested, both in certain work setting and parallel home settings (e.g., 
carpentry, construction, home remodeling projects). Further, the use of objects that can be counted or 
manipulated (coins, beans) can shed more light on the ability of low-ability or low-literacy individuals to 
handle certain everyday situations involving simple quantitative information. (The use of other more 
sophisticated objects, such as a computer spreadsheet, of course can also fit under the assessment of 
numeracy, but in PIAAC is taken under the framework of the Problem Solving domain). 

89. The use of a calculator, ruler, or objects such as coins is in principle desired in an assessment of 
adult numeracy skills. Yet, actual implementation in a large-scale assessment carries both psychometric, 
operational, and cost implications when there is a need to test thousands of adults in their homes in 
multiple countries in a standard and efficient manner. For these reasons, in PIAAC's first cycle respondents 
are able to use hand-held calculators as well as paper (printed) rulers that have both metric and imperial 
measurements. However, it is not possible to use other types of objects such as country-specific coins due 
to the heterogeneity in this regard. 

4.2 PIAAC approach to assessment 

90. The PIAAC assessment design involves using a household survey methodology which assumes 
that overall testing time per respondent is around 60-80 minutes. During this time, there is a need to 
administer a short core test (screener), present direct assessment items in one or more of the different 
competency domains (Fiteracy, Numeracy, Problem-solving), and collect information about the 
respondents’ background and various correlates of interest via the background questionnaire (BQ). The 
design for the direct assessment in PIAAC incorporates computer-based measures of competencies when 
possible, yet also use written ("paper and pencil") assessment booklets for adults who are unfamiliar or 
uncomfortable with computers. 

4.2.1 Adaptive testing. 

91. To increase assessment efficiencies, the direct assessment is administered to the majority of 
respondents via a computer platform, TAO, which uses a computer-based adaptive testing process. The 
adaptive testing process means that tasks (i.e., stimuli and questions about them) are shown on a computer 
screen, the respondents answer on the computer, and their answers are automatically (immediately) scored 
as correct or incorrect, without human judges or coders being involved in interpretation of responses. This 
automatic scoring is essential because adaptive testing is based on the cumulative performance on tasks; at 
various points during the assessment TAO decides, based on decision rules stored by the computer 
program, what additional assessment tasks (at higher or lower difficulty levels) to select from a pool of 
assessment items for presentation to the respondent. 

92. The key advantage of an adaptive testing scheme is that is can achieve the best estimate of each 
respondent’s ability level, using a smaller number of assessment items than in a traditional test design 
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where respondents have to answer all questions included in the test, from easiest to most difficult. Thus, 
adaptive testing can enable deeper and more accurate assessment of respondents ability level, while 
reducing response burden and the chance respondents will face many tasks which are above their ability 
level and hence cause frustration. 

93. However, the assessment of numeracy in the first cycle of PIAAC is constrained in several ways 
because of the assessment design. Firstly, the overall testing time per respondent does not allow inclusion 
of extended problems or lengthy simulations of complex authentic numeracy tasks, although it is 
recognised that ability to solve complex or extended numeracy problems is an inherent part of the 
numeracy competency. In order to cover all facets of the numeracy construct in the limited time available, 
the use of a larger number of short tasks is prescribed. 

94. Secondly, the need to score all responses automatically limits the type of assessment tasks that 
can be used. While the TAO system allows respondents to provide an answer in several different modes 
(e.g., numeric entry, clicking on an area of the screen, choosing from pull-down menus), in its present 
stage of development it cannot accept most types of free-form text-based answers because of the huge 
possible diversity in how respondents may enter their answers. The limitations stem from the difficulty to 
automatically code (i.e., designate an answer as correct or incorrect) free-form responses in dozens of 
languages while accommodating various grammatical and syntactical structures, as well as overcoming 
typing mistakes which are naturally expected when people type text into a computer. Examples are when 
respondents: 

• write number ranges or estimates which have multiple mathematically-equivalent 
representations, such as "a quarter", "0.25", "1/4", "1 in 4", or “around five to six”, "1.00 
to 6.00"; 

• provide explanations of how a certain result was reached (“subtracted six from the sum of 30", "I 
did 30 - 6"); 

• describe their interpretation of given information such as in a simulated media statement; 

• write justifications for their answers, or list arguments supporting their conclusions. 

95. As a result of the restrictions discussed above, certain types of numeracy tasks, especially those 
involving interpretation or evaluation/analysis with communication responses, receive only partial or slight 
coverage in the first cycle of PIAAC. With this in mind, as part of the search for ways to circumvent 
somewhat the limitation on text-processing in the computer-based testing environment, in a few numeracy 
questions respondent may be asked to provide an explanation for a response by choosing from pre- 
designed encapsulated texts, so as to simulate the way a person provides a justification for an answer in 
real life. However, such experimental solutions are partial at best and have their own limitations. It is thus 
hoped that in future cycles of PIAAC, some of the current technical limitations will be resolved, allowing 
for broader coverage of more aspects of the numeracy construct. In addition, it should be noted that 
respondents uncomfortable or unfamiliar with using computers are directed to the paper-based branch of 
the assessment. While the tasks they encounter in that portion of the test are mostly duplicates of 
computer-based items, a few do require respondents to communicate interpretations or explanations of 
their reasoning about some tasks via free-form written answers, thus helping to expand the coverage of the 
numeracy construct slightly beyond what is possible in the computer-based assessment alone. 
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4.2.2 Item pools and scaling. 

96. The intent of PIAAC is to have its results linked to previous international adult assessments. 
Therefore, the general PIAAC design requires that 60 percent of the literacy and numeracy tasks will come 
from item pools used in ALL and lALS. These former items serve as linking items, and in addition new 
items were developed for PIAAC that can fit the computer-based adaptive testing requirements and 
constraints. Overall, the items for numeracy assessment are expected to enable reporting of respondents' 
performance in a manner similar to the one used in ALL and lALS, which scaled raw ability scores in the 
range 0-500, but mainly focused on reporting performance on five ability levels with the following 
tentative boundaries: 

• Level 1 : raw score of 0 - 225 (lowest level) 

• Level 2: raw score 226 - 275 

• Level 3: raw score 276 - 325 

• Level 4: raw score 326 - 375 

• Level 5: raw score 376 - 500 (highest level) 

4.3 Principles for assessing numeracy in PIAAC 

97. The development of numeracy assessment for PIAAC has been based on a number of general 
principles or guidelines listed below. These principles reflect the cumulative literature on large-scale 
assessment of mathematical skills and adult numeracy (Gal et ah, 2005; Gillespie, 2004; Murat, 2005), and 
various background documents and positions prepared as part of the planning of PIAAC (e.g.. Gal, 2006; 
Jones, 2006; Murray, 2006; Tout, 2006), the general ideas listed earlier in this section, as well as the 
known technical limitations in the first cycle of PIAAC: 

a. Items should cover as many aspects as possible within each of the four facets of the 
numeracy competency. Items should require the activation of a broad range of skills and 
knowledge included in the construct of numeracy, as portrayed in the conceptual 
framework depicted in Table 1. Specifically, all four areas of mathematical content, 
information, or ideas (Lacet 3 in Table 1) should be covered, with relative proportions 
as follows: 

• 30% Quantity & Number 

• 25% Dimension & Space 

• 20% Patterns, Relationships, Change 

• 25% Data & Chance 

b. Items should aspire to maximal authenticity and cultural appropriateness. Tasks should 
be derived from real-life stimuli and pertain to all types of contexts or situations (i.e., 
everyday life, work, societal, further learning) that can be expected to be of importance 
or relevant in the countries participating in PIAAC. Item content and questions should 
appear purposeful to respondents across cultures, although it must be acknowledged 
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that in a large-scale assessment such as PIAAC, not all items and contexts can he 
personally familiar to all adults within any one country, let alone across all countries . 

c. Items should have a free-response format, to the extent feasible by the computer 
platform used for administering the direct assessments in PIAAC. Items should he 
structured to include a stimulus (e.g., a picture, drawing, visual display) and one or 
more questions, the answers to which the respondent communicates via the modes 
available within TAO, primarily: numeric entry, click, highlight a region of the 
stimulus, usage of various pull-down menus. (Text entry is limited to very specific 
words or sometimes a simple number due to the concerns listed above regarding the 
inability to score text entries with keying/typing errors, and the presence of multiple 
ways to express the same mathematical entities in words and/or numbers). In addition, 
items allowing a free -form response will be used in the paper-and-pencil portion of 
PIAAC, which some respondents will take, allowing for some expansion of possible 
responses, beyond those presently afforded within the computer platform. 

d. Items should spread over different levels of ability. Items should span the range of 
ability levels anticipated within PIAAC participants, from low-skilled individuals 
(which are of interest in countries where policies and educational programs may be 
earmarked for low-skill populations), all the way to those with advanced competencies. 

That said, it should be recognised that the need to reduce the number of items to be 
administered in any one domain has led designers of past assessments (lALS and ALL 
for adults, PISA for school students), as well as in PIAAC, to include few very easy 
items (i.e., items at level 1) and few very hard items (i.e., items at Level 5). Instead, 
respondents will be classified as at Level 1 if they could not do well on Level 2 tasks. 
Likewise, those classified at Level 5 will be those who performed well on Level 4 items 
and on the few real Level 5 items. It follows that a more detailed assessment of the 
specific skills that Level 1 respondents have requires a separate diagnostic assessment, 
such as the assessment of component literacy skills planned for PIAAC. 

Given the above, to enable the adaptive testing process reach a an efficient estimation 
of respondents' ability levels, the following distribution of items at the different 
difficulty levels is likely to be sought for constructing the numeracy item pool for the 
Main assessment, based on the results of the field-test (pilot) in 2010: 



• Level 1: 

• Level 2: 

• Level 3: 

• Level 4: 

• Level 5: 



10% of the numeracy items 
25% of the numeracy items 
30% of the numeracy items 
25% of the numeracy items 
10% of the numeracy items 



e. Items should represent the different response types. However as mentioned already, 
certain types of numeracy response types, especially those requiring the use of 
interpretation, evaluation, analysis and communication, will receive only partial or 
slight coverage in the first cycle of PIAAC due the the computer based assessment 
platform and its constraints at this stage. Therefore for PIAAC the response types of 
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Interpret, Analyse/evaluate and Communicate have been collapsed into a single 
response type. It is hoped that in future cycles of PIAAC, some of the current technical 
limitations will be resolved, allowing for better coverage of more aspects of the 
numeracy response facet. Given the above, the following distribution of items requiring 
the different types of response types will be sought for constructing the numeracy item 
pool for the main PIAAC direct assessment: 

• 10% Identify, locate or access 

• 50% Act upon, use: order, count, estimate, compute, measure, model 

• 40% Interpret, evaluate/analyze, communicate 

f. Items should vary in the degree to which the task is embedded in text. Some items 
should be embedded in or include relatively rich texts, while others should use little or 
no text. This distribution aims to reflect the different levels of text involvement in real- 
world numeracy tasks, as well as reduce overlap with the literacy scale. 

g. Items should be efficient. To allow for coverage of many key facets of the numeracy 
competency, the inclusion of a large number of diverse stimuli and questions will be 
needed. However, in light of testing time constraints, the use of short tasks is 
necessitated, precluding items that can simulate extended problem-solving processes or 
that require a lengthy open-ended response. 

h. Items should be adaptible to unit systems across participating countries. Items should 
be designed so that their underlying mathematical demands are as consistent as possible 
across countries, regarding language and mathematical conventions. For example, items 
should be designed so that different currency systems or different systems of 
measurement (metric or Imperial) could be applied to the numbers or figures used. 

Items should retain equivalency with respect to their mathematical or cognitive 
demands after being translated. 

4.4 Factors explaining item/task complexity 

98. In planning an assessment, it is of course desirable to be able to understand what it measures. 
Assessment designers assume that when engaged with the assessment items (including tasks, questions, 
stimuli, etc), respondents activate cognitive processes and rely on stored knowledge and learned skills 
which are part of the construct being measured. Thus, differential performance levels can be accounted for 
by the underlying cognitive knowledge bases and other enabling processes. It follows that it is useful to have 
a theoretical model or set of assumptions regarding what factors cause certain tasks to be harder or more 
complex than others, so that the assessment results can be correctly interpreted. A model or scheme of factors 
affecting task complexity can also help when linking the assessment results to possible social (or educational) 
interventions, i.e., point to the skills that are lacking and have to be further developed in the population 
(Brooks, Heath, & Pollard, 2005). 

99. Prior seminal work by Kirsch and Mosenthal (e.g., Kirsch, 2001) and earlier projects has pointed 
to several key factors which account for task difficulty when considering arithmetic items or items 
involving text comprehension, primarily readability, type of match, plausibility of distractors, operation 
specificity ('transparency'), and type of calculation and number of steps. The Kirsch & Mosenthal work has 
informed the design of assessment tasks for lALS and other surveys, and the interpretation of their results. 
In designing the ALL numeracy scale, the ALL Numeracy team has attempted to advance the Kirsch and 
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Mosenthal complexity scheme and develop tentative assumptions regarding factors which affect difficulty 
of multiple types of new tasks introduced to measure the numeracy construct which were heyond those 
encompassed hy the more focused construct of Quantitative Literacy in lALS. Examples are items 
involving percents, knowledge of measurement and spatial reasoning, statistical concepts, and so forth. 

100. The developers of the Mathematical Literacy scale for PISA (2006) also recognised multiple 
factors affecting item difficulty, such as the kind and degree of interpretation and reflection required hy the 
problem, the kind of representation skills required, or the kind and level of mathematical skill required, 
e.g., single-step vs. multi-step problems, or more advanced mathematical knowledge, complex decision- 
making, and problem solving and modeling skills, or the kind and degree of mathematical argumentation 
required. Eurther factors that are assumed to affect difficulty both in PISA, ALL and other surveys relate to 
the degree of familiarity with the context, and the extent to which tasks require reproduction of known 
procedures and steps or present novel situations requiring non-routine and perhaps more creative 
responses. It should be noted that the PISA description of complexity factors seems quite compatible with 
that of ALL, although some of the terminology is different, and published PISA reports do not explain in 
detail how it was used to guide the design of specific items. 

101. The complexity scheme for numeracy used in ALL (Gal et ah, 2005) has been instrumental for 
the item development and scale construction stages of that study, especially in that it helped to evaluate in 
advance if items will span different difficulty levels. Given that PIAAC's numeracy assessment is founded 
on the principles developed for ALL and that the PIAAC numeracy assessment scale uses over two dozen 
linking items used in ALL, the ALL complexity scheme has been adopted as an analytic tool for item 
development and interpretation for PIAAC as well. Lurther details about this scheme are provided in 
Appendix 1, which is adapted from Gal et al. (2005). 



PART 5: DIFFERENCES BETWEEN PIAAC'S NUMERACY AND RELATED SCALES 



102. To gain a better understanding of what is measured in the numeracy domain in PIAAC, it is 
important to discuss the differences between numeracy and related constructs targeted in international 
assessments, such as quantitative literacy and mathematical literacy. As will be seen, the differences are 
more a matter of degree rather than these constructs being totally different from each other - after all in one 
way or another they all pertain to some aspects of people's mathematical knowledge. Further, it must be 
pointed out that the differences emerge in more clarity when looking not at definitions (i.e., the conceptual 
level) but at their operationalisation (i.e., the assessment scale design, constraints on assessment, and the 
actual assessment tasks). 



5.1 Adult assessments 

103. Let us first examine some conceptions developed in international surveys of adult skills. A 
framework developed by Kirsch and Mosenthal (see Kirsch, Jungblut, & Mosenthal, 1998) to describe 
adults' literacy skills, including aspects of adult's quantitative skills, has been widely implemented in 
multiple national and international assessment projects, most recently the International Adult Literacy 
Survey (lALS; see Statistics Canada and OECD, 1996, 1997). The lALS framework made use of three 
literacy scales — Prose Literacy, Document Literacy, and Quantitative Literacy — to operationalise its 
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conception of literacy. The PIAAC domain of numeracy is most closely related to the Document Literacy 
(DL) and Quantitative Literacy (QL) scales, defined as follows. 

DL: The knowledge and skills required to locate and use information contained in various 
formats (including job applications, payroll forms, transportation schedules, maps, tables, 
and graphics). 

QL: The knowledge and skills required to apply arithmetic operations, either alone or 
sequentially, to numbers embedded in printed materials ( such as balancing a check book, 
figuring out a tip, completing an order form, or determining the amount of interest on a 
loan). 

104. QL tasks as well as some DL tasks have addressed important aspects of people’s mathematical 
knowledge and skills. Lor example, DL tasks required respondents to identify, understand, and interpret 
information given in various lists, tables, charts and displays; this information sometimes included 
quantitative information, such as numbers or percents. QL tasks required respondents to apply arithmetical 
operations learned mostly in elementary grades; these tasks did not require respondents to cope with other 
types of mathematical information (e.g., measurements, shapes) or with information whose processing 
does not require comprehension of text. In addition, tasks used in both QL and DL scales called for a 
limited range of responses, i.e., exact computations or specific types of interpretations. Such tasks and 
responses are important by themselves, yet they represent only a subset of the much wider range of tasks 
and responses that are typical of many everyday and work tasks, such as sorting, measuring, estimating, 
conjecturing, or using models (e.g., formulas). Thus it can be concluded that QL and DL cover a subset of 
the dimensions and ideas captured by numeracy for PIAAC. 

5.2 School-age assessments 

105. In the context of international assessments of school-age students, a central construct is 
mathematical literacy. This term first appeared as part of the second TIMSS, where students in their final 
year of secondary schooling (usually 12* grade) were assessed not only on mathematical knowledge but 
also on “mathematics literacy” (Mullis et al, 1998:43), defined as follows: 

Mathematics literacy items address number sense, including fractions, percentages, and 
proportionality. Algebraic sense, measurement, and estimation are also covered, as are 
data representation and analysis. Several of the items emphasise reasoning and social 
utility. A general criterion in selecting the items was that they should involve the types of 
mathematics questions that could arise in real-life situations and that they are 
contextualised accordingly. 

106. This definition illustrates the connection between conceptualisation and operationalisation as two 
building blocks of the construct. But it can also be seen that the construct of mathematics literacy was 
defined through its facets, without there being a general definition. 

107. PISA (2006) defines mathematical literacy as follows: 

Mathematical literacy is an individual’s capacity to identify and understand the role that 
mathematics plays in the world, to make well-founded judgements and to use and engage 
with mathematics in ways that meet the needs of that individual’s life as a constructive, 
concerned and reflective citizen. 

108. This definition shows some overlap and consistency with the conception of numeracy used in the 
present framework as well as with broader conceptions of literacy as adopted by lALS, ALL, and PIAAC. 
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Table 2: Mathematical content areas covered by PIAAC and PISA 



PIAAC 


PISA 


- Quantity & number 


- Quantity 


- Dimension & shape 


- Space and shape 


- Data & chance 


- Uncertainty 


- Pattern, reiationships, & change 


- Change & reiationships 



109. Table 2 lists the four key content areas covered by the numeracy assessment in PIAAC, in 
comparison with the four content areas of mathematical literacy covered in PISA. While using somewhat 
different terminologies, the two schemes of mathematical content refer to quite similar domains overall and 
point to conceptual compatibility between PIAAC and PISA. 

110. There are several additional similarities between PISA’s Mathematical Literacy assessment of 
school students of age 15 and PIAAC’s Numeracy assessment of adults, such as: 

• PISA examines how students of age 15 cope with tasks embedded in four contexts: 
personal, educational/occupational, public and scientific, which are quite similar to 
those contexts discussed in section 2.4.1 above regarding adult numeracy 

• The PISA assessment framework for mathematical literacy highlights PISA’s focus on 
real-world problems, moving beyond the kinds of situations and problems typically 
encountered in school classrooms, and demanding “the ability to apply those skills in a 
less structured context, where the directions are not so clear, and where the student 
must make decisions about what knowledge may be relevant and how it might usefully 
be applied” (PISA, 2006; p 72). 

• The description of the four types of responses (i.e., identifying, locating or accessing; 
acting upon, using; interpreting; and communicating), bear some resemblance to the 
five steps in the mathematisation cycle posited in the PISA framework as involved in 
solving real-life problems. These steps involve, among other things: identifying 
mathematical information in a situation after trimming away non-essential elements in 
reality, solving a mathematical problem while switching between representations and 
using formal operations or technical language as needed, linking mathematical solutions 
and making sense of them in light of the real situation, or explaining results. 

• The enabling processes that support numerate behavior, discussed above, relate to the 
eight key competencies described in the PISA framework, such as thinking, reasoning, 
modeling, problem-posing and problem-solving, communication, representation, and 
using symbolic and technical language. 

111. While there are numerous similarities between the framework for PIAAC numeracy and the 
PISA framework for mathematical literacy, differences also emerge, in part due to the different 
environments for which these frameworks have been developed. PISA relates to school-based populations, 
and while it is interested in students’ performance on real life problem, a basic underlying assumption is 
that the performance is to be based on skills and dispositions acquired in a schooling context. As a result, 
descriptions of students’ desired actions or underlying cognitive processes in operation are couched in a 
school-based environment. Indeed, an arguable cursory examination of published PISA items shows that 
there is room to further examine the extent to which items use realistic contexts and stimuli. Some items 
use formal symbolism that reflect an expectation for formal knowledge of what was taught in schools, yet 
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such knowledge is less (or not) available to adults who have been out of formal school environment for 
years. 

1 12. It can be assumed that adults, much more so than 15-year olds, have personal experiences and 
ways of coping with everyday situations which are different than those of school-age students. Hence, the 
types of responses envisioned of adults tested in PIAAC, and the explanations for underlying enabling or 
causative factors (such as “literacy skills”) are not couched in a “mathematical problem-solving” culture. 
For this reason, task realism in PIAAC assessment may play a somewhat different role than in PISA. As a 
result, the numeracy framework for PIAAC, while being informed by the established literature on school 
mathematics, certainly goes beyond it, and at times uses different terms and ideas that are based on 
additional literatures. 

113. Note should also be taken of the role of literacy (in the narrow, technical sense of reading and 
writing) in PISA's mathematical literacy. Despite the inclusion of the term “literacy” in “Mathematical 
Literacy”, the PISA mathematical assessment does not seem explicitly interested either in tasks where 
mathematical information is embedded in text, or in the influence of literacy skills on mathematical 
performance, described earlier in this framework. In creating the PISA Mathematical Literacy scale, 
relatively little effort appears to have been taken to control the literacy content of tasks; as a result, very 
high correlation were obtained between reading literacy and mathematical literacy scale scores; this in turn 
was one (but not the only) contributing factor towards a situation whereby countries scoring high on one 
scale usually (but not always) scored high on other scales. In contrast, the designers of the ALL numeracy 
scale sought to reduce the literacy demands of at least some of the numeracy items. This in turn contributed 
to numeracy scores having lower correlations with Document Literacy scores in ALL, compared to the 
relatively high correlations between Quantitative and Document Literacy scores in lALS. 

5.3 Further issues in comparing large-scale assessments 

1 14. The comparison of assessment frameworks, even in a single domain such as mathematics, can be 
a complex undertaking. This was demonstrated by a recent project of the National Center on Education 
Statistics (NCES) in the United States, which aimed to compare the mathematics frameworks and items for 
three large scale assessments, the National Assessment of Education Progress, TIMSS, and PISA (Neidorf, 
Binkley, Gattis & Nohara, 2006). The project involved comparisons of frameworks along multiple 
dimensions or topics, such as the mathematics content and process skills to be assessed, the main content 
areas included and the set of subtopics covered in each, calculator use policy, and so forth. The analysis of 
the commonalities and differences between the three assessments also included comparisons of hundreds 
of items, in terms of the mathematics content covered, performance expectations for different grade levels, 
the complexity of different tasks (e.g., the extent to which they require application of routinised versus 
novel approaches), cognitive processes underlying different items, item formats, and item contexts. This 
project required the work of a panel of a dozen experts over several days. 

115. One of the general specifications for PIAAC's numeracy is that there would be conceptual 
continuity with Mathematical literacy as viewed by PISA. While the two constructs are related they should 
not be viewed as identical, for reasons explained above. Each construct has somewhat different operational 
implications at the level of assessment design and item content. For example, PISA items are somewhat 
different than PIAAC items due to differences in authenticity of tasks, the contexts from which tasks are 
drawn (and the role of technology and tools in them), and in particular the increased use of formal 
mathematical symbolisations, which many adults are normally not familiar with after leaving school. 
Further, numeracy implies the need for greater attention to dispositional aspects of the competence. 

116. It follows from the above that the PIAAC numeracy scale is compatible and shares much 
common ground with PISA’s mathematical literacy scale, as well as with ALL’s numeracy and to some 
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extent with lALS QL and DL scales. Yet, the nature of the commonalities and differences cannot he fully 
analyzed at this stage without further investment in a more fine-grained analysis and some systematisation 
of terminology, thereby creating simpler bridges between PIAAC and PISA. Eventually, the meanings that 
can be attached to assessment results from PIAAC in the area of numeracy, and the degree of overlap 
between PIAAC’ s numeracy and PISA’s mathematical literacy, depend not only on the conceptualisation 
of numeracy in PIAAC, but probably more so on a host of other factors discussed earlier. It must be kept in 
mind that an analysis of commonalities and differences between PIAAC's and PISA's mathematical scales 
cannot be conducted only on the conceptual level - it needs to also consider the characteristics of the actual 
assessment tasks. Such characteristics include task realism, density of text in stimuli, and so forth, as well 
as the constraints on assessment (i.e., what questions can be asked, what coding is possible, etc) imposed 
by the features of the computer platform for implementing PIAAC's direct assessments. 



PART 6: SUMMARY AND FURTHER REFLECTIONS 



1 17. Given the increasing need for adults to continuously adapt to changing citizenship and workplace 
demands (European Commission, 1996; Cohen, O'Donoghue & EitzSimons, 2000), the assessment of 
numeracy is essential so that countries have a solid basis from which to design social interventions and 
effective lifelong learning opportunities that can improve competencies (OECD, 2006). Accordingly, 
numeracy has been conceptualised in this document as a broad construct that pertains to adults’ level of 
coping with a diverse range of numeracy tasks couched in real-world contexts. 

118. This document has reviewed literature and research regarding competencies (part 1), perspectives 
on the complex meanings associated with numeracy and related constructs, and the numeracy demands 
faced by adults in various contexts (part 2). Based on this background, later subsections in part 2 presented 
a definition of numeracy as a basis for developing the assessment scale for PIAAC. These theoretical 
foundations then served as the basis for discussing (part 3) facets of numerate behavior related to Contexts, 
Responses, Mathematical content/information/ideas, and Representations, each with several components. 
This document has also emphasised the importance of assessing dispositions and practices as an integral 
part of the numeracy competence, given that they affect performance on numeracy tasks and can correlate 
with various variables of interest. Later parts elaborated on assessment principles and design constraints 
inherent in PIAAC's assessment plan and computer-based platform (part 4), and examined commonalities 
and differences between PIAAC and other scales (part 5) in order to help readers understand how to 
interpret PIAAC results pertaining to numeracy and connect them with results regarding mathematical 
literacy (PISA) or Quantitative literacy (lALS). 

1 19. At the conceptual level, the definition of numeracy in this framework is in general compatible 
with the ALL conceptualisation, yet introduces some advances of a modest scope that go beyond what 
existed in ALL. These changes bring the conceptualisation of numeracy presented in this document closer 
to the conception of "literacies in the information age" employed by PIAAC, while allowing for more 
compatibility with the PISA definition of mathematical literacy, as desired by OECD. The changes in the 
conceptualisation of numeracy were also introduced with a long-range view in mind, to enable the 
accommodation of new types of numeracy-related tasks and demands faced by adults in the information 
age in future cycles of PIAAC, while maintaining a common conceptual definition across assessment 
cycles. 
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120. It should be recognised that some modest overlap exists between key constructs measured in 
PIAAC, i.e., Document literacy, Numeracy, and Problem-Solving in Technology-Rich Environments, on 
the conceptual level but also separately at the scale or item level. In particular, numeracy includes as one of 
multiple sub-areas the ability to read and interpret quantitative information in graphs and tables, yet this is 
also subsumed as one part of Document Literacy. The existence of such an overlap is sensible and 
expected. After all, many real-world tasks, such as interpretive tasks which do not require the manipulation 
of numbers, or tasks involving quantitative statements embedded in text, require adults to integrate the use 
of numeracy and literacy skills (Kirsch et ah, 1993; Gal, 2002a). Likewise, numeracy involves the ability 
to solve multi-step or extended problems couched in a technology context, not just short simple tasks, and 
some numeracy-related content is encountered when adults handle problems involving financial, 
scheduling, or other everyday tasks. Hence tasks chosen for the Problem-solving scale may also touch on 
numeracy topics. The presence of overlap between scales, however, should not be a cause for concern or a 
reason for restricting the conceptual definition of numeracy. The construct of numeracy stand on its own 
and reflects an authentic part of the adult world. In addition, one must also consider that “assessment drives 
instruction”: Excluding some areas from the conceptualisation of numeracy to reduce overlap or inter-scale 
correlations may limit what delivery systems in, e.g., educational or workplace training contexts, will 
target when trying to develop desired competencies. 

121. As emphasised earlier, what is measured by a numeracy assessment scale (direct assessment) and 
the associated BQ items (regarding numeracy-related practices, attitudes, beliefs, and so forth) is 
determined not only by a conceptual framework describing numeracy and its facets and enabling factors - 
it is also determined by an assessment framework. Such a framework describes how the general 
conceptualisation of numeracy is operationalised and manifested in the nature and range of tasks used in 
the actual assessment, and specifies what limitations may be created due to the mode of task administration 
and of scoring. 

122. Overall, an assessment is a complex dynamic system that combines conceptual and technical 
elements. Eventually, the outcomes of an assessment, and the reliability, validity, and usefulness of the 
findings and their interpretations, are influenced by criteria and values placed by the assessment designers, 
and by the many choices they make about task design and methodology. The implementation of the 
numeracy conceptual framework in the first assessment cycle of PIAAC is influenced by practical 
limitations of the computer-based platform, which prohibited the use of certain types of open-ended, 
communication and evaluative type items. In addition, the need to cover a broad construct within the short 
timeframe available for assessment, a typical situation in large-scale household-based assessments, has 
forced the use of short tasks with machine scorable response formats, further limiting the use of extended 
problems. It is hoped that these limitations will gradually lessen as greater sophistication in computer 
platforms will enable more flexibility in task design, question-posing and response formats in future cycles 
of PIAAC. 
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ANNEX 



FACTORS AFFECTING COMPLEXITY OF NUMERACY ITEMS 



lA. This appendix describes a scheme of factors that account for the difficulty of different numeracy 
assessment tasks. The scheme was developed for the Adult Literacy and Life Skills Survey by the ALL 
Numeracy team. The following text is copied (with permission) from the ALL Technical Report, see Gal et 
ah, 2005. This scheme was found useful to inform item development, i.e., help in the creation of items that 
spread over a range of difficulty levels. Results from the ALL pilot study showed that predicted difficulty 
of items used by the scheme described below was highly correlated with observed difficulty (r = 0.79). 
Because of the recursive nature of the testing of this scheme (e.g., the same individuals wrote the scheme 
and rated the complexity of items), caution should be exercised in further interpretive use of the present 
version. While further validation is needed, the scheme in its current state nonetheless appears to also be a 
possible useful tool for interpretation of testing results. 

Previous research on task complexity 

2 A. In lALS, three factors were found to be the principal components of task difficulty (regarding 

literacy or text-based tasks): plausibility of distractors, type of match required, and type of information 
required. The difficulty of the Quantitative Literacy tasks appeared to be a function of several other factors: 

1 . The particular arithmetic operation required to complete the task 

2. The number of operations needed to perform the task 

3. The extent to which the numbers are embedded in printed materials 

4. The extent to which an inference must be made to identify the type of operation to be 
performed (i.e. problem transparency; see below) 

3A. The lALS QL difficulty factors overall fit those used in large-scale assessments of mathematical 
skills (with children), which often make use of three or four factors: 

1 . The mathematical concepts involved: number systems and number sense, spatial and 

geometrical topics, functions and algebra, chance/statistics topics, etc. Concepts that are 
related to topics taught in lower grades are considered easier. 

2. The complexity of operations: addition, subtraction, multiplication, and division, as well as 

dealing with whole numbers, with decimals, and with percents. Operations that are related 
to topics taught in lower grades are considered easier. 

3. The number of operations: one-step problems are considered easier than multi-step problems. 
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4. Problem transparency: This factor is sometimes relevant; it refers to the extent to which the 

problem situation includes clearly identified numbers or entities and the extent to which it is 
clear what operations or actions to perform. To the extent that these are not clear or 
transparent, respondents have to extract needed information by applying comprehension 
and inference strategies, making the task more complex. 

4A. There are other adult-related assessment projects on which to draw to develop the levels of 
complexity. Both the Essential Skills Research Project and the Applied Numeracy sub-test of the Work 
Keys test battery (American College Testing, 1997) use a two-factor model of complexity in their 
description of numeracy levels. The first factor, “operations required;” is seemingly straightforward and 
refers to the difficulty of operations called for. However, this is complicated by the level of difficulty of the 
numbers being manipulated: computations that include fractions and decimals are usually more difficult 
than those with whole numbers. 

5A. The Essential Skills model spells out two sequences of complexity on this factor: Operations and 
Translation of information (sometimes called 'problem transparency'). 

Operations 

1 . Only the simplest operations are required and the operations to be used are clearly specified. 

Only one type of mathematical operation is used in the task. 

2. Only relatively simple operations are required. The specific operations to be performed may 
not be clearly specified. Tasks involve one or two types of mathematical operation. Eew steps 
of calculations are required. 

3. Task may require a combination of operations or multiple-applications of a single operation. 
Several steps of calculation are required. (More advanced operations may call for 
multiplication or division.) 

4. Tasks involve multiple steps of calculation. 

5. Tasks involve multiple steps of calculation. Advanced mathematical techniques may be 
required (e.g., percents, ratios, proportions). 

Translation (Problem Transparency) 

1. Only minimal translation is required to turn the task into a mathematical operation. All the 
information required is provided. 

2. Some translation may be required or the numbers needed for the solution may need to be 
collected from several sources. Simple formulae may be used. 

3. Some translation is required but the problem is well defined. 

4. Considerable translation is required. 

5. Numbers needed for calculations may need to be derived or estimated; approximations 
may need to be created in cases of uncertainty and ambiguity. Complex formulae, 
equations or functions may be used. 



45 




EDU/WKP(2009)14 



6 A. Two considerations prompted us to question the appropriateness of using mathematics-related 

frameworks (from Essential Skills or elsewhere) as the sole source for development of a complexity 
scheme for items assessing adults’ ability to cope with real-world numeracy tasks. Eirst, effective coping 
with many real-world quantitative problems depends upon people’s ability to make sense of and interact 
with different types of texts. This is hardly recognised by the Essential Skills model. Hence, it was 
essential to add difficulty factors that acknowledge the inherent links between literacy and numeracy, quite 
similar to those used in lALS. 

7 A. Another, albeit a more restricted consideration, is that the ordering of complexity of tasks by the 

type of operation performed may not be as clear with adults as it may be with children. Such ordering in 
school-based assessments is predicated on traditional school curricula, where more advanced topics are 
learned at higher grades. However, adults are known to use a lot of invented strategies, perhaps more so, 
and more efficiently so, than children. Multiplication or division problems, which can prove relatively hard 
for some young people, may be solved by seemingly simpler strategies, such as by repeated addition or 
repeated subtraction; complex numbers may be broken down in ways that ease mental load, and so forth. In 
addition, adults’ familiarity with everyday contexts, such as with monetary entities, facilitates their 
performance with some seemingly advanced concepts. Eor example, specific benchmark values of 
fractions and percents, such as 1/2, 1/4, 50%, or 25%, are familiar to many people; as a result, they may be 
easier to manage than expected, violating curriculum-based ordering of difficulty. Hence, an overall 
complexity level has to be used, in order to weight these “inconsistencies” in ordering of difficulty levels 
proposed in other schemes. 

Complexity factors in the ALL survey 

8A. The above literature review suggests that a framework of factors affecting the complexity of 
numeracy tasks should not only address factors related to the numerical and textual aspects of tasks, but 
should also address other issues. It should treat separately the number of operations and the type of 
operations from the type of mathematical (or statistical) information to be processed, which may involve 
numbers explicitly but also other types of mathematical information. In so doing, the desired framework of 
complexity factors should take into account the broad scope of the definition of numeracy, i.e., reflect the 
variation within contexts, the range of mathematical ideas/content, the types of possible responses, and the 
types of representations that cut across adult life contexts. 

Table 1 A. Complexity Factors - Overview 



Aspects 


Category 


Range 


Textual aspects 


1. Type of match/problem 
transparency 


Obvious/explicit to 

embedded/hidden 




2. Plausibility of distractors 


No distractors to several 
distractors 


Mathematical 

aspects 


3. Complexity of Mathematical 
information/data 


Concrete/simple to 

abstract/complex 




4. Type of operation/skill 


Simple to complex 




5. Expected number of operations 


One to many 



9A. With the above considerations in mind, five key factors have been identified that are predicted to 
affect, separately and in interaction, the difficulty level of numeracy tasks to be used in the ALL survey. 
These five "complexity factors" are outlined in Table lA and are organised in two sets: two factors that 
address mainly textual aspects of tasks, and three factors that address the mathematical aspects of tasks. 
These five factors are listed separately for clarity of presentation, but in actuality are not independent of 
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each other and do interact in complex ways. Each factor is examined in some detail helow, followed hy a 
later subsection that describes the calculation of an overall complexity level for each item, taking into 
account all five factors. 

Type of Match/Problem Transparency. 

lOA. This is a combination of the factor of Problem Transparency outlined above, and of an lALS 
factor called Type of Match. Problem Transparency is a function of how well the mathematical 
information and tasks are specified and includes aspects such as how apparently the procedure is set out, 
how explicitly the values are stated, etc. Type of Match refers to the process that a respondent has to use to 
relate the requested action in the question to the information in the task or text, which can range from a 
simple action of locating or matching to more complex actions that require the respondent to perform a 
number of searches through the information given. This measure of complexity for a numeracy task 
incorporates the degree of text embeddedness of the mathematical information. 

1 lA. In easy tasks, the type of information (e.g., numerical values) and the operations needed are 
apparent and obvious from the way the situation is organised. In more difficult ones, the values must be 
located or derived from other values; the operations needed may have to be discovered by the performer, 
depending on his or her interpretation of the context and of the kind of response expected. As well, 
numeracy situations may involve text to varying degrees, and this text may be of different degrees of 
importance. There may be a situation where there is little or no text. Some situations may involve pure 
quantitative information that is to be interpreted or acted upon with virtually no text or linguistic input. In 
other words, the performer derives all the information needed to respond from the objects present in the 
situation or from direct numerical displays. 

12A. At a higher level, some textual or verbal information may be present alongside the mathematical 
information. The text can provide background information about the problem situation, or some 
instructions. Eor example, a bus schedule, cooking instructions, and a typical school-type word problem all 
involve some text and some numbers. Still other situations would be heavily text-based or may not involve 
any numbers or mathematical symbols at all, just plain text. The task will contain mathematical or 
statistical information that a person needs to understand and, in some cases, act upon, but it will be much 
less transparent. It may be heavily embedded in dense text or may require using information from a 
number of sources within or even outside the text/task. 

13 A. This factor requires that a task will be analyzed in terms of the questions: How difficult is it to 
identify and decide what action to take? and How many literacy skills are required? 

Plausibility of distractors. 

14A. This variable is literacy related, even though it can involve mathematical components. In general, 
literacy tasks are easiest to process when there are no plausible distractors in the text, that is, there is no 
other information in the text that meets any of the requirements of the task. At higher levels of difficulty, 
tasks can involve irrelevant information both within the question as well as within the text. In terms of 
mathematical information, a low level of plausible distractors would mean that no other mathematical 
information was present apart from that requested, making the numbers or data required easy to identify. 
At a higher level, there may be either some other mathematical information in the task (or its text) that 
could be a distractor, or the mathematical information given or requested could occur in more than one 
place. A higher level of complexity could also mean that outside information (e.g. the knowledge of a 
formula) may be needed to answer the question. 
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15 A. This factor requires that a task will he analyzed in terms of the questions: How many other pieces 
of mathematical information are present?, and Is all the necessary information there? 

Complexity of Mathematical Information. 

16A. Some situations present a person with simple mathematical information, such as concrete objects 
(to be counted), simple whole numbers, or simple shapes or graphs. At lower skill levels, the information 
will be more familiar, whereas at higher levels, the information may be less familiar. Situations will be 
more difficult to manage if they involve more abstract or complex information, such as very large or very 
small numbers, unfamiliar decimals or percents, information about rates, or dense visual information, as in 
a diagram or complex table. 

17A. This factor requires that a task will be analyzed in terms of the question: How complex is the 
mathematical information that needs to be manipulated or managed? 

Type of Operation/Skill. 

18A. Some situations require simple operations, such as addition or subtraction, or simple 
measurement (e.g., finding the length of a shelf), or recognition of shape. These are usually easier to 
analyze mathematically than situations that require multiplication or division, and than situations that 
require using exponents. While the difficulty of recognizing and carrying out the operation implied by a 
situation (be it additive, multiplicative, etc.) has direct bearing on task complexity, there may be exceptions 
that occur when alternative approaches are obvious. There are some tasks that combine both interpretive 
and generative skills and may involve a deeper conceptual understanding than merely carrying out a 
procedure. Other more complex tasks may involve an explanation of one’s reasoning. The interpretation of 
information appearing in graphs, for example, becomes more complex if comparisons, conjecturing, or 
“reading beyond the information given” is required. 

19A. This factor requires that a task will be analyzed in terms of the question: How complex is the 
mathematical action that is required? 

Expected Number of Operations. 

20A. Tasks that require acting upon the mathematical information given may call for one application 
(step) of an operation, or for one action (e.g., literal reading of information in a table, or measurement). 
More complex tasks will demand more than one operation, which may be the same or similar to one 
another, such as the steps involved in multiple passes on the data or text. Still more complex tasks are those 
that involve the integration of several different operations. 

21A. This factor requires that a task will be analyzed in terms of the question: How many steps and 
types of steps are required? 

Overall Complexity Level 

22A. It is possible to estimate the overall difficulty level of a specific item by first scoring the item on 
each of the five factors of complexity, according to the levels described in Appendix 2, and then summing 
together the scores for each factor. Figure 1 below explains the process; Appendix 2 describes each level of 
the five factors in detail. The total summary score can range between 5 (easiest) and 19 (most difficult). 

23A. The estimation process outlined in Figure 1 suggests that each factor has a separate contribution 
to an item's overall difficulty or complexity. However, it can be hypothesised that as tasks become more 
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complex, actual performance on items may increasingly depend not only on each factor by itself, but also 
on the interplay or interaction between them. Hence, the computational process suggested in Figure 1 can 
provide only approximate information about an item's anticipated difficulty level. 

24A. Further, the difficulty of a task cannot in some cases be predicted without taking into account 
characteristics of the person who interacts with the task. The same task may be more difficult for some 
individuals and less difficult for other individuals, depending on factors such as their familiarity with the 
context in which a task is situated, knowledge of formal mathematical notations, background world 
knowledge, as well as general literacy, problem-solving, and reasoning skills. For example, it could be 
predicted that a task that involves the composition of a fertilizer would be more difficult for an urban 
apartment dweller than for a rural farmer whereas a task that uses a bus schedule would be more difficult 
for the farmer. For the above reasons, the prediction of the difficulty of a task in isolation of detailed 
knowledge about the respondent himself can only be an estimate. 

25A. Despite the above limitations, the scheme of complexity factors developed for numeracy 
assessment in ALF comprises a theoretical contribution. It provides a conceptual basis for predicting the 
different levels of complexity of a broader range of items well beyond those involving arithmetic 
operations only. Indeed, this scheme was highly correlated with observed difficulty (r = 0.79). Because of 
the recursive nature of the testing of this scheme (e.g., the same individuals wrote the scheme and rated the 
complexity of items), caution should be exercised in further interpretive use of the present version. While 
further validation is needed, the scheme in its current state nonetheless appears to also be a possible useful 
tool for interpretation of testing results. 
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Figure 1 A. Complexity Flowchart 



Factor 
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SCORING FOR EACH OF THE COMPLEXITY FACTORS 



Complexity Factor 1. Type of match/Problem transparency 

How difficult is it to identify and decide what action to take? How many literacy skills are required? 


Score 1 


Score 2 


Score 3 


In the question and the 
stimulus, the information, 
activity or operation required: 

- is clearly apparent and 
explicit — and all required 
information is provided 

- is specified in little or no text, 
using familiar objects and/or 
photographs or other clear, 
simple visualisations 

- is about locating obvious 
information or relationships 
only 

- closed question — not open- 
ended 


In the question and the 
stimulus, the information, 
activity or operation 
required: 

- is given using clear, simple 
sentences and/or 
visualisations where some 
translation or interpretation 
is required 

- is located within a number of 
sources within the 
text/activity. 

- fairly closed question 


In the question and the 
stimulus, the information, 
activity or operation required: 

- is embedded in text where 
considerable translation or 
interpretation is required 

and/or 

- may need to be derived or 
estimated from a number of 
sources within or outside the 
text/activity 

and/or 

- the information or action 
required is not explicit or 
specified 

- more complex, open-ended 
task 



Complexity Factor 2. Plausibility of distractors 

How many other pieces of mathematical information are present? Is all the necessary information 
there? 


Score 1 


Score 2 


Score 3 


- no other mathematical 
information is present apart 
from that requested — no 
distractors 


- there is some other 
mathematical information in 
the task that could be a 
distractor 

- the mathematical 
information given or 
requested can occur in more 
than one place 

- may need to bring to the 
problem simple information 
or knowledge from outside 
the problem. 


- other irrelevant mathematical 
information appears 

- mathematical information 
given or requested appears in 
several places. 

- necessary information or 
knowledge is missing, so 
outside information or 
knowledge needs to be brought 
in 
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Complexity Factor 3. Complexity of mathematical information/answer required 

How complex is the mathematical information that needs to be manipulated? 


score 1 


score 2 


score 3 


score 4 


score 5 


Context 

Based on very concrete, real 
life activities, familiar to most 
in daily life. 


Based on common, real life 
activities. 


Based on real life activities, 
but less often encountered. 


Based on real life 
activities but unfamiliar to 
most 


Based on abstract ideas 
or unfamiliar activity in a 
context new to most. 


Quantity 

Whole numbers to 1 ,000 
Fractions, decimals, 


- large whole numbers 
including millions 

- other benchmark 
fractions, like V 3 and V 10 

- common decimals, like 
0.1 , 0.25 to 2 decimal 
places 

- common whole number 
percents, like 25% and 

1 0 %. 


- large whole numbers 
including billions 

- other fractions 

- decimals to 3 decimal 
places (other than 
money) 

- all whole number 
percents 


- negative integers 

- all remaining fractions, 
decimals and 
percentages 


- all remaining types 
of rational (and 
some irrational) 
numbers including 
directed numbers 


percents 

- benchmark fractions (V 2 , 
V 4 , %) 

- decimal fraction for a half 
only (0.5) and equivalent 
as a percentage (50%) 


Pattern and relationshio 

- very simple whole 
number relations and 
patterns 


- simple whole number 
rates and ratios 

- whole number relations 
and patterns 


- rates and ratios 

- relations and patterns 
including written everyday 
generalisations 


- complex ratios, 
relations, patterns 

- simple formula 


- formal mathematical 
information such as 
more complex 
formulae, knowledge 
of relationships 
between dimensions 
or variables, etc 
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Measures/ 

Dimension/SDace 

- standard monetary 
values 

- common everyday 
measures for length 
(whole units) 

- time (dates, hours, 
minutes) 

- simple, common 2D 
shapes 

- simple localised maps or 

plans (no scales) 


- everyday standard 
measures for length, 
weight, volume , 
Including common 
fraction and decimal 
units 

- common 3D shapes 
and their representation 
via diagrams or photos 

- common types of maps 
or plans with visual 
scale Indicators 


- other everyday measures 
(area Included) Including 
fraction and decimal 
values 

- more complex 2D and 3D 
shapes, or a combination 
of 2 shapes 

- area and volume 
formulae 

- common types of maps or 
plans with ratio type 
scales 


- all kinds of 
measurement scales 

- complex shapes or 
combinations of 
shapes 




Chance/Data 

- simple graphs, tables. 


- graphs, tables, charts 


- graphs, tables, charts 


- complex graphs. 




charts with few 


with common data 


with more complex data 


tables or charts 




parameters and whole 


Including whole number 


(not grouped data) 


Including grouped 




number values 


percents — whole 


- more complex data or 


data 




- simple whole number 


number scales In Is, 
2s, 5s or 10s 


statistical Information 
Including common 


- complex data or 




data or statistical 


- data or statistical 


average, chance and 


statistical Information 




Information In text 


Information Including 


probability values 


Including probabilities. 






whole number percents 


- scales: more complex 
whole number, fractional 
or decimal 


measures of central 
tendency and spread 
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Complexity Factor 4. Complexity of Type of operation/skill 

How complex is the mathematical action that is required? 



score 1 


score 2 


score 3 


Communicate 

no explanation - a 
single simple 
response required 
(orally, or in writing) 


- no explanation - a simple 
response required (orally, or 
in writing 


- simple explanation of a (level 
1 or 2) mathematical process 
required (orally, or in writing) - 


Compute 

- a simple 
arithmetical 
operation (+, x, 
with whole numbers 
or money 


- calculating common fraction, 
decimal fraction and 
percentages of values 

- using common rates (e.g. 
$/lb.); time calculations; etc 

- changing between common 
equivalent fraction, decimal 
and percent values, including 
for measurements e.g. V 4 kg = 
0.250kg 


- more complex applications of 
the normal arithmetical 
operations such as calculating 
with fractions and more 
complex rates, ratios, 
decimals, percentages, or 
variables 

- simple probability calculations 


Estimate 


- estimating and rounding off 
(when requested ) to whole 
number values or monetary 
units 


- estimating and rounding off to 
requested number of decimal 
places 


Use formuia/ modei 


- evaluating a given formula 
involving common operations 

(+, X, -) 
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score 4 



score 5 



- explanation of a (level 3) 
mathematical process 
required (orally, or in 
writing) 


- complex, 
abstract and 
generative 
reasoning or 
explanation 
required 


- applications of other 
mathematical operations 
such as squares, square 
roots, etc 


- more advanced 
mathematical 
techniques and 
skills e.g. 
trigonometry 


- making a contextual 
judgment re whether a 
found answer is realistic or 
not and changing the 
answer to the appropriate 
correct rounded (but not 
necessarily 

mathematically correct) 
answer. 




- developing/creating and 
using straight forward 
formulae 

- using strategies such as 
working backwards or 
backtracking (e.g. 15% of 
? = $255) 


- generative 
reasoning 

- using and 
interpreting 
standard 
algebraic and 
graphical 
conventions and 
techniques 
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Measure 

- knowing common 
straight forward 
measures 

- naming, counting, 
comparing or sorting 
values or shapes 


- visualizing and describing 
shapes, objects or geometric 
patterns or relationships 

- making and interpreting 
standard measurements 
using common measuring 
instruments 


- using angle properties and 
symmetry to describe shapes 
or objects 

- estimating, making and 
interpreting measurements 
including interpolating values 
between gradations on scales 

- converting between standard 
measurement units within the 
same system 


- calculating measures of 
central tendency and 
spread for non-grouped 
data 

- converting between non- 
standard measurement 
units within the same 
system 

- counting permutations or 
combinations 


- converting 
between 
measurements 
across different 
systems 


Interoret 

- locating/identifying 
data in texts, graphs 


- reading and interpreting data 
from texts, graphs and tables 


- interpolating data on graphs 


- generating, organising, 
graphing non-grouped 
data 


- graphing 
grouped data 

- calculating 


and tables 


- following or giving straight 


- calculating distances from 


- extrapolating data 


measures of 


- orientating oneself 
to maps and 
directions such as 
right, left, etc 


forward directions 


scales on maps 


- reading and interpreting 
trends and patterns in data 
on graphs, including 
slope/gradient 


central tendency 
and spread for 
grouped data 
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Complexity Factor 5. Expected number of operations 

How many steps and types of steps are required? 


score 1 


score 2 


score 3 


one operation, action or process 


application of two or three steps, 
the same or similar operation, 
action or process 


integration of several steps 
covering more than one different 
operation, action or process 
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